Amen
This is (one of) the exact same point(s) I've been making for weeks.
A pioneering American community ISP is telling customers that Netflix should spend more time improving its technology, more money on its network – and less energy on lobbying in Washington DC. BSD developer and author Brett Glass founded Lariat.net in Wyoming in 1992, making it one of the first ISPs in the world. He told The …
Hold on a second. These ISP can start griping when I see every one of them that grips has a viable competitor in their geographic area. These broadband companies milk it for all it is worth and don't put any investment back into the system to improve service. Fact is we need 3 wired service providers for each area and then we'll start getting them to STFU.
"…. don't put any investment back into the system to improve service."
We live in the forest, low density multiacre lots. A few weeks ago our ISP rolled out fiber optic to the area including going literally the extra mile to our neighbourhood. They've invested tens of millions here and tens of millions there and so on.
I'll acknowledge that YMMV.
>>> The "BSD developer" who gave us the bikeshed!
And the freebsd security advisory!
The post office told me I buy too many stamps off them...
The taxi drivers told me I hire their taxis too often...
The restaurant told me I eat there too often...
The cinema hated I watched every film...
The ISP capped my bandwidth...
Luxuries are not a resource that is depleted, they are something people pay for. If a company turns down our custom, we take our money else where. Don't complain we spend it with your competitor when you stop the service you agreed to provide for the cash.
If your not an Internet Service Provider, than advertise as something more fitting like "media filtering service" and see if people pay for it...
The post office charges you per letter, the taxi company charges you per mile, the restaurant charges you per dish (even the all you can eat buffet gets mad if you leave food on your plate), and the cinema charges you per film. But the Internet provider charges a flat rate whether you push 500 KB a month or 500 GB, simply because it's always been that way. The "solution" to this problem, unfortunately, may turn out to be metered pricing (and no, I'm not looking forward to it either.)
> But the Internet provider charges a flat rate whether you push 500 KB a month or 500 GB, simply because it's always been that way. The "solution" to this problem, unfortunately, may turn out to be metered pricing (and no, I'm not looking forward to it either.)
Except "bits" aren't in limited supply, nor are the accumulated use of such what is causing congestion on the tubes. The problem is concurrent bandwidth use exceeding the ISP's capacity, which metered-by-the-bit internet caps do not remedy. For example, if everyone burns through their allotment of bits in the first week, when that new Netflix show comes on, there's still going to be congestion, even though all the customers are well within their limit of bits.
Hence the only logical method of such tiered pricing is using bandwidth bands (10Mb/s, 30Mb/s, etc.) as is currently done by most ISPs already.
The fundamental issue is that ISPs - all of them - grossly oversell their available bandwidth/throughput. It's the reason ISPs sell "up to xMb/s broadband", and historically this was generally good enough for most people. Then internet video streaming came along, and customers started demanding ISPs make good on the inflated numbers they've been spouting for years.
No. No the ISP does not. My ISP charges me for the bandwidth. I get some in my monthly bill, and the rest I pay as I go.
Even "flat rate" is pay per GB in disguise. It's the ISP who decide to offer fate rate, so their decision if they can or cannot deliver such a service. They must advertise/provide a different service that is all they can provide/sell.
The solution is metered pricing, because like other utilities it charges people for what they use. Can you image the waste if people paid a fixed charge for water, electricity & gas regardless of consumption? I find running water makes a nice background noise, but I don't leave a tap on because water is precious and it is costs per litre used.
Metered pricing will encourage ISPs to build faster networks, because you can download more data meaning that they can charge more.
Speed tier pricing makes each jump more expensive. A 10Mbps connection is about 3.24TB/month, 100Mbps is 32.4 TB/month, 1Gbps is 324TB/month. I shouldn't be expected to pay for someone else's 24/7 4HD netflix streaming habit, when I only want my fast connection to video conference for a couple of hours each month.
" Can you image the waste if people paid a fixed charge for water, electricity & gas regardless of consumption?"
Um...You said 'tap' instead of 'forcet' which implies you are in the UK - yet you still don't know that the majority of properties here have unmetered water supplies?
In my 40+ years, every property I have lived at in the UK has had a fixed water charge, and funnily enough I've never left taps on for no good reason!
Similarly, I won't leave live video streaming running on my unmetered internet connection if I'm not using it.
If you think about it, I'm sure you'll find many things you don't waste even when cost isn't a factor.
"according to wikipedia (I know), UK metered supplies covered 33 of households and leads to a 5-15% reduction in usage."
Fair enough, though I've never known anyone to leave taps on for the hell of it.
Though I do know some people with meters don't flush the toilet...
" I know I am much more aware of water use on a metered supply than I was ona flat rate supply"
Aware, maybe, but I'm curious to how you use less. (or rather, what you used before than you don't now).
Not being a git, I'm genuinely clueless - other that watering the garden, maybe washing the car less, but as for the indoor use...?
The problem is that it is not about the amount of water delivered, but the capacity to deliver water.
You might be able to fill a swimming pool using a garden hose, but it will take days to do so. You probably would prefer to pay for a sufficient supply to be able to fill it in hours rather than days - same amount of water, totally different demands on infrastructure.
Bandwidth is the requirement for the end user. Bits used is of concern to the provider for their transit cost, together with bandwidth capacity to meet the peaks of demand.
Bandwidth is a fixed cost, transit cost is a variable one.
Not in this instance. The Netflix issue is congestion between the ISP peers and/or Netflix, not between the customer peers the ISP is serving. Hence the point about allowing the ISP to cache the data.
Yes, I'd generally prefer metered pricing because I think it solves the the freetard skimmer problem, but this isn't a problem it would fix. I'll particularly note this as a FIOS customer who also uses Netflix. The rest of my streaming is fine, it's ONLY Netflix which has issues with buffering, stutter, or sometimes even not being able to display information about a tv show/movie I want to watch.
The solution is metered pricing, because like other utilities it charges people for what they use. Can you image the waste if people paid a fixed charge for water, electricity & gas regardless of consumption?
See, this is where the water & power utility analogy breaks down. Water, electricity and gas (CNG or LPG) are finite resources. The utility company has to buy that from someone else to give it to you, the more you use, the more stuff the utility company has to buy (electricity, gas, water, whatever).
Data, however, is sent through fixed "pipelines". Your ISP only pays for a fat pipe, with a fixed data rate and sometimes variable pricing on certain data rates (say, base rate covers up to 10 Mb/s, then you get charged per Mb/s extra up to the physical limit for that pipe being 30 Mb/s), but the thing is: they are paying the same as you are charged, by data rate. So it shouldn't be an issue if you're using 5%, 50% or 100% of your allocated data rate all the time because that is what the ISP sold you in the first place!!!! If ISPs want to get better data rates, they should upgrade their uplink pipes, increase pricing for home subscribers or reduce advertised speeds. The days of 50:1 contention ratios are over.
"The "solution" to this problem, unfortunately, may turn out to be metered pricing (and no, I'm not looking forward to it either.)"
Not sure why you were downvoted so much, and although I don't agree with what you're saying, I come from the "pay per bit" side and it is a big mess.
Our "all you can eat" buffet meals are never that, the fine print has some moving goal post the inflicts limits. There's nothing inherently wrong with that except: the bulk of our software comes from the US were buffet is order of the day. We're stuck with apps that eat data like there's no tomorrow. But there IS a tomorrow, when the bill arrives and we're fucked.
Especially when it comes to media, even here, there is a definite push to internet radio, internet TV, internet everything - all coming from a place where data is free for all. Even if we're willing to pay for this monstrosity, it STILL doesn't fix the pipes. Our beloved NBN is nearly a stillbirth unless you're lucky to be living in the middle of nowhere (great for testing - no-one pays attention to small townfolk when things go wrong), and the pipes in the city areas where they're really needed are in limbo - GovCo has given up and is trying to sell it off.
Bottom line is, we're still fucked, because on one side you're getting one group who's telling us "this" is the way it's supposed to be, on the other side, no bastard wants to pay for it.
nice to hear another vantage point, and I am sure they have a good point. Netflix, stop it.
But it still doesn't detract from the aims of what *should* be boring network companies, and trying to find a way to treat data as "special" in some way.
Gigabit everywhere, then let's argue about "special" data...
P.
Ok, I'll try. I work at a very small ISP around the world but try to follow what is happening in the West.
Big problems with the U.S. Netflix situation are geographical and "telco-political" (yeah I made that up, deal with it). Very large country, largely sparsely populated where these micro/nano-ISPs (often wireless/WISP) provide only viable service that could described as "approaching broadband". Other option would be the incumbent and bad DSL over bad copper, often at ridiculous prices. To someone not from US it may seem unbelievable that even in middle of urban area, say Silicon Valley, there might be only one provider who can service you with residential fibre or high-speed cable.
The other issue is importance of private peering over Internet Exchanges. In Europe large amount of interconnection between providers is done at IXPs. At US there are very few IXPs and providers have their own private peering arrangements. Add to this the geographical/competetive situation and politics of peering by the big (at least a local monopoly really) players and what is left is the 1000 mile dark fibre from the rural WISP to Netflix.
While that's all true, there's something else at work here. For a US resident, I live in a sweet spot for ISP cable competition: the DC metropolitan area. Comcast and Verizon BOTH have the area wired for high speed service, plus there are a number of other players in the area. Even so, Verizon is not delivering Netflix adequately. Until a few months ago, the same was true with Comcast. Which doesn't mean those problems won't need to be solved to get reliable high speed streaming to rural customers, only that even if they had that service Netflix would still be an issue. In fact, if Brett Glass is correct, the issue will be more pronounced in high density metropolitan areas than low density rural ones, simply because there will be a higher likelihood of parallel streams in the high density area.
another voice of reason. Too many people think Netflix does no wrong.
(myself I killed my account when they had the first price hike because I realized how little I was actually using it at the time - have not seen any reason to re-subscribe to it or any other streaming media service)
Netflix told us that, if we wanted to improve streaming performance, we should pay $10,000 per month for a dedicated link, spanning nearly 1,000 miles, to one of its "peering points" – just to serve it and no other streaming provider.) It then launches misleading PR campaigns against ISPs that dare to object to this behavior.
Well, Netflix is selling the product that needs that fat pipe so really they should foot the bill. On the other hand you could see it as the ISP footing the bill to better server its customers. However if the ISP foots the bill for the link then it should be able to do whatever it wishes with it. I don't see why Netflix would have any say what the ISP does with a link the ISP pays for.
This kind of ties in with the recent threads about the Netflix/Verizon spat on El Reg. There were discussions about where the "fault" lies and I did raise the question about peering arrangements and who actually should pay for what.
This kind of ties in with the recent threads about the Netflix/Verizon spat on El Reg. There were discussions about where the "fault" lies and I did raise the question about peering arrangements and who actually should pay for what.
I recall that you certainly did.
My take:
IF
Netflix 'foots the bill' for the 'fat pipe' to the ISP, then Netflix has every right to say how it is used
BUT
if the ISP 'foots the bill' for a fatter pipe, then Netflix can go out and fuck itself
HOWEVER
if they agree to 'split' the bill, then that may be a reasonable commercial arrangement
It all depends on who blinks first, or gets their nuts squeezed by a regulator first.
It's sounds like he covered that to be honest. Obviously Netflix puts a different spin on it, but it really says the same thing. "ISPs can do this either by free peering with us at common Internet exchanges," but if that isn't nearby you'll have to pay for the link yourself "or can save even more transit costs by putting our free storage appliances in or near their network." i.e. host their power-hungry servers for free.
It's not too hard to imagine that neither option is very appealing for a small ISP.
Might not be appealing, but does it provide the service you promised to your customer? I'm paying my ISP for the connectivity, not Netflix.
If the ISP wants to modify our contract to include a Netflix peering surcharge, because the cost of that connection is disproportionate to the rest of the service (use whatever method of determining typical non-Netflix user you want) that's fine and I'll evaluate the value of that new contract. After which I'll decide whether paying the charge or canceling Netflix. BUT I expect you'll be adding similar charges for other heavy broadband usages such as video Torrent downloads as well.
"Might not be appealing, but does it provide the service you promised to your customer? I'm paying my ISP for the connectivity, not Netflix.
If the ISP wants to modify our contract to include a Netflix peering surcharge, because the cost of that connection is disproportionate to the rest of the service (use whatever method of determining typical non-Netflix user you want) that's fine and I'll evaluate the value of that new contract."
Exactly! I have sympathy for the ISPs, but the way the model is structured, it isn't Netflix pushing lots of data into ISPs, it's the ISP customers requesting/pulling/downloading the data as allowed under the terms of their contract.
If an ISP no longer finds the 'all you can eat' (but hopefully not many of you will) model sustainable, then they need to reevaluate that model. Whether it's to place restrictions, up the prices generally, or charge more for higher usage (and it should be based on the data amount, not who's supplying it) then so be it.
As it is, they are charging customers for the right to download that data, and then when they do, they want to charge the provider of that data too.
Yeah, I can see how as Netflix gets popular it seriously impacts on your business model, but that's not Netflixs fault for supplying data your users have paid to be able to receive. The flaw is your underselling, and it's only your problem.
You'd have the same problem if all your users simply downloaded the same amount of data from loads of other random sites.. You going to try and charge every site for the data your users download from it?
P.S. I'm referring to the basic bandwidth issue - the CDN/cache/local server/hosting aspect is another big can of worms
That's the problem - Netflix doesn't work with small ISPs for openConnect, and even if they did, Netflix doesn't pay for the hardware or pipe for it.
Netflix would be totally on the good-fight side of this if they offered openConnect to everyone (and maybe kicked in some of the operating costs). As it stands, it leave a bad taste in your mouth when small ISPs get ignored like this - it's another strike against them that they shouldn't have to deal with.
Personally, I think they should use services like Akamai, which were built for exactly this situation, but it's possible that the content providers don't allow it. I suspect that IS the case, or openConnect wouldn't need to exist in the first place.
Glass knows very much what he is talking about - obviously much more than most of those making comments here. After all, he actually does it for a living.
The "Free" peering that the page you posted mentions requires $10K+/month connection, and the "storage appliances" draw significant amounts of power and generate a lot of heat (10 Amp @ 120 VAC each)
The storage appliances are not caching servers, they don't download content on demand and store it if a second customer requests it. They pre-download and store popular content so that it is there IF a customer requests it, and act as their CDN to push content to other servers. They require 10 GB Ethernet connections, often consume over 7 GIGABITS of internet bandwidth even when a customer isn't streaming video, and download a minimum of 7 Terabytes of data nightly for the catalog refresh.
In short, they are extensions of Netflix's infrastructure. Servers that any other company would pay to have co-located at a facility. Netflix tries to bully their way in for free co-location *AND* make the ISP pay to haul the traffic.
Since Netflix has a limited number of peering points, even if you connect to a Tier 1 peer if you aren't geographically close the connection between the peering points (the Internet backbone, outside of the ISP's network) get congested and cause viewing problems even the the ISP's connections to the backbone are nowhere near saturated.
"...Netflix tries to bully their way in for free co-location *AND* make the ISP pay to haul the traffic."
Agreed. My working assumption is that Netflix are being manipulative. I'll grant that the USA and elsewhere are plagued with subpar ISPs, but that doesn't really change the apparent fact that Netflix is manipulting the anger of the unwashed masses through a campaign of misinformation.
Thankfully my ISP is recently stellar (fiber, 175 Mbps), and I've no use for the brain Pablum on offer from Netflix. So I'm not directly involved in this mud slinging festival of deception.
Traffic is traffic is traffic - why is it different for the ISP piping Netflix to the customer than any other traffic?
As far as I remember, the price for GB was 2-3 cents at the interchange, so all those 2 - 3 TB of data that Netflix is pushing to the servers at the ISP is $40 - 90 dollars a day. I am not sure how much power they would draw, but let's put some made up numbers, let's say 20 servers at 200 W each, which should be 4 kW/h, about 50 cents/hour, or 12 dollars a day.
So for 100 bucks a day, an ISP can host Netflix? Is $3000 a massive amount for even a small ISP with a few thousand customers? I don't know, I am asking.
Once you have the traffic on your network, it's not an issue. The reason ISPs like Verizon don't like it is because the offset of data coming in to data going out on those links is very lopsided, and that messes up transit agreements and costing.
A balanced link from one ISP to another is consider a wash financially - both sides have equal skin in the game so it's in everyone's interest to keep the data flowing. With (for example) cogent's link to verizon sending 10 times as much to Verizon than it gets back from them, it means that one side has more power than the other, and you're seeing Verizon throw it around. Is that a good thing for the net? Of course not, but it makes Verizon subscribers more likely to give in and get pay-per-view from them instead of getting the same movie from netflix.
Lately, I've been using itunes rentals a lot more for PPV stuff - it costs a buck less than my ISP charges, and because it's downloaded instead of streamed, you get a much better viewing experience all around if bandwidth is an issue. I would really like to see Netflix switch to an expiring-download model for their net-viewable content - it would be no less secure than a stream (i.e. neither is secure at all if you know what you're doing and are determined to snag a copy), it would stop a lot of these problems, and the viewing experience would be better.
Netflix's software is really infuriating in how it buffers - even if you purposely pause the video to let it build up some content, at some point it will throw the buffer on the floor anyway. Just give us the option to let the whole bar fill if we want!
Let me get this straight. Customers are asking Lariat whether Lariat will give them decent access to Netflix in exchange for paying a monthly subscription that is probably considerably more than the customer is paying Netflix, and it's Netflix's fault that the answer is no?
It's not clear whether Mr Glass is losing customers because they're switching to other ISPs that can deliver what he can't, or whether he's the only game in town, and he's complaining that his users can't get decent Netflix access because the population of Wyoming is too spread out to get decent bandwidth cheaply. But Netflix business model is working for Netflix - it doesn't have to work for lariat.net
Netflix is helping Lariat.net by providing content that is enticing enough that people want to sign up to Lariat.net so that they can get access to Netflix. The money that that new user pays splits about 80-20 in Lariat.nets favor.
There's no question that delivery of broadband to more widely dispersed populations, further from exchange points has different economies than delivering service in denser urban environments. But as I said, that's Lariat.net's business model, not Netflix's business model.
By the way, I should have pointed out that I don't even have a Netflix subscription.
By design, it's impossible to cache netflix content at the ISP level.
Well, not impossible as such (there are tools to capture the stream), but not "just plonk some hardware at the border and cache this" easy. And it would probably get you into copyright trouble if you a commercial enterprise even tried it.
Not impossible to cache encrypted traffic, you just need to know the keys (which a netflix controlled cache would do)
There is no copyright trouble for a cache that doesn't make available stuff that wasn't available anyway on the network (at least not in the US, DMCA safe harbour regs took up a not insignificant amount of my employment a few years back)
I put quite alot of effort into designing such things for a previous employer - because ISP level caching is just so damned stupidly obvious - at least it is if you've worked with ISPs in the past (as had my previous employer).
Particularly for things like netflix, which are likely to have short time high value items (just released shows etc)
So a company successfully creates a product that customers want. An ISP has customers that want that product but the ISP can't provide enough bandwidth to supply it. So then the ISP blames the company creating the service? So any successful company that provides content on the Internet should pay any ISP that has a customer that wants that content. Sounds like the ISP wants to charge two people for one service. Now, the remark that he, the ISP, can get customers to an uplink at reasonable speed but the pipes are slow from there might or might not be legitimate. But that has nothing to do with net neutrality.
On the contrary, this has everything to do with net neutrality, or at least the case as it is presented in public. These are exactly the issues it is claimed net neutrality will resolve. Something to think about the next time you see an article about the FCC toying with the idea of re-writing the rules to implement it.
"Netflix generates huge amounts of wasteful, redundant traffic and then refuses to allow ISPs to correct this inefficiency via caching"
Holleywood won't let Netflix let the ISP's cache the content and see Openconnect as has been pointed out. Those inefficiencies cost Netflix money too Mr. ISP . . .
Holleywood won't let Netflix let the ISPs cache the content, so Netflix created openconnect servers for the ISPs. Those ineffinecies cost Netflix money too.
The last thing that we want is a provider has to pay the ISPs. If that is the case small ISPs that serve small audiances are likely to be leftout due to economics. Why would Netflix bother trying to reach a few thousand possible subscribers in sparcely populated areas.
Until I thought, hang on. I pay for a 180mbps Internet connection. If I choose to fill it with Netflix then I choose to fill it with Netflix. If I choose to fill it with Spotify then I choose to fill it with Spotify (or Pi OS downloads, or any other downloads). If I breach your cap then my bad (if I agreed to it) but if not then why should you throttle data which I have already paid for? Don't have the capacity then don't sell it.
I'm sure there are arguments against Netflix doing bad things, but if the data throughput is less that I have paid for then it's less than I have paid for.
You miss the point.
If all the traffic is Netflix there is a problem with the connection between ISP and Netflix as in this case there is no affordable peering for the ISP.
The content provider has to pay for their connection to "internet".
Your sub is only paying for connection to ISP and the ISP's normal peering costs.
The whole net neutrality debate is too simplistic.
It's not quite that simple.
Imagine Netflix content is delivered by the lorryload to your home from a Netflix factory on the other side of the country.
You pay your local council to maintain streets and local roads that connect your home to the motorway network.
Netflix pay their local council to maintain the streets and local roads that connect them to the motorway network.
The question here is about who pays to maintain the motorway network. It's full of lorries carrying Netflix content. Netflix are effectively arguing that your local council should pay to build a new motorway linking them to the Netflix factory, or at least to the motorway junction very near their factory; or alternatively your council should pay for the land and infrastructure and ongoing costs required to have Netflix build a new factory in your town. Your local council are unsurprisingly arguing that Netflix should pay to build and maintain the motorway as far as the motorway junction near your town, or at least pay the rent and infrastructure costs for Netflix to build that factory in your town.
In most similar disputes, traffic flows more or less equally in both directions, so the answer is normally for both parties to split the cost between them. However, as this particular motorway will need many more lanes heading from the Netflix factory to your town than in the other direction, the argument is not so clear cut.
Disputes like this are going to run and run...
Not a good analogy.
A better one would be: I am paying company X to deliver me a product from the warehouse of company Y.
Now X, instead of only charging me, wants to charge Y as well.
And when I pay for shipping, I don't care that X has to pay for toll roads, gas and trucks. That is their business.
"Imagine Netflix content is delivered by the lorryload to your home from a Netflix factory on the other side of the country.
You pay your local council to maintain streets and local roads that connect your home to the motorway network."
Your analogy is flawed, because whilst you mention roads, you don't talk specifically about cars!
All geek analogies should be based on cars! It's the law!
"All geek analogies should be based on cars! It's the law!"
NOT when it's a bulk transport analogy. Cars don't fit that analogy as they're not considered a bulk transport vehicle, so it HAS to be lorries. That's law, too. Even the Net Neut debate uses lorries in its diagrams."
I concur with this observation, but nothing else in your post.
To me the consumer, I'm paying you the ISP for my connection to the internet. Period. As the ISP you are assuming all of the risks associated with providing that service WHEN I want it for whatever service I want to use and managing the costs.
They can't. The content MAKERS hold the trump cards: the actual content. Their content, their copyright, their rules. They can refuse to publish and keep things locked tight in theaters and BluRays or simply roll their own. The only reason Netflix is so popular is that its content range is so broad. The content makers can kill that pretty quickly, though.
This geezer is offering wireless connections, so no line rental.
http://www.lariat.net/rates.html
Prices seem fairly steep by UK standards though, even given the lack of line rental. I personally could get by on his cheapest tariff as I don't do streaming. Others might need much more.
How does the 'last mile' work in US? Lower population density = harder I imagine.
The reason it's so bad is that the US is so BIG. Wiring up tiny little Great Britain isn't exactly a picnic, but at least the distances aren't so bad. But in the US, you have people from coast to coast, and unless it's a big city, the ROI just isn't there unless the communities in need can sweeten the deals with exclusivity contracts. For many small communities, it's the price of admission: either bind themselves to contracts or go without. It's like that for other utilities, too, like natural gas, since there's a significant infrastructure investment required just to reach those communities.
Use the electricity supply cables?
I may have totally misunderstood this of course as I have no local knowledge.
http://en.wikipedia.org/wiki/Broadband_over_power_lines
http://www.washingtonpost.com/blogs/the-switch/wp/2013/09/17/how-chattanooga-beat-google-fiber-by-half-a-decade/
http://en.wikipedia.org/wiki/Utility_cooperative
ISPs should not really know what data is streaming into their portion of the network or where it originates. What is being communicated is none of the business of the carrier be they carrying postal mail or transmitting video. In this respect there is a net neutrality issue because in order to charge someone like NetFlix an ISP would have to be prying into private communications.
Ultimately, the users of the network should pay for the network. Both the customer downloading content and the provider (such as NetFlix) streaming the content are 'users'. They should both pay to send or receive whatever they are sending or receiving to the backbone. They are 'peers' in this sense. NetFlix pays their bandwidth providers to push data up to the backbone and their customers pay *their* bandwidth providers to pull the bandwidth down from the backbone.
It makes little sense for the same traffic to be constantly traversing the network. It *should* be cached as a matter of course. However, if NetFlix must encrypt streams for each individual recipient, it will not actually be the same data that is being sent.
If a company like NetFlix is abusing the network by forcing the same enormous volumes across the network over and over, they should be made to bear some financial cost associated with any malfeasance such as refusing to allow caching. Until they bear the cost of some of the inefficiencies they introduce, they will have little incentive to correct their behavior. That being said, we should err on the side of maximizing neutrality with respect to sender/receiver and the nature of the data being moved.
It seems to me that we come to problems like this because bandwidth is limited and much of that limitation is an enforced artificial scarcity to support the old-fashioned revenue models of the communications cartels. Where I live, Bell Canada still charges the unwary as much as $0.91 per minute for a long distance call within the province. [http://www.bell.ca/Home_phone/Long_distance_rates]. Even with our semi-crippled network infrastructure that is better than a 100000 per cent mark up; pretty good if you can get it.
We are still treating EM spectrum, cable and telephone lines as if what they are carrying is pinned to how it is being carried. This has carved up bandwidth inefficiently and resulted in a lack of competition among the different modes of transport. Both result in higher costs for bandwidth.
We need to get everyone on board to create enormous transparent backbone networks that are essentially public assets that are essentially free to use and to remove regulations artificially propping up differences between transport that no longer apply. We also need to have a conversation guided by people we trust. There is much confusion about all this and it is because the waters have been muddied by people who simply don't understand the network attempting to work against disinformation being supplied by ones who do understand it but have a vested interest in the confusion allowing them to stifle competition and charge more for things than they are worth.
The confusion sown by both the genuinely confused and the network cartels means that we can never have a sensible conversation about prioritizing bandwidth. Some things, such as real-time responses to timing signals, keystrokes, etc require low latency. Some things, such as voice communications, require QOS so that there are no interruptions sufficiently long to interrupt communication. Some things require lots of bandwidth, some require very little. Some traffic, such as text messages require very little in terms of quality. EMail does not suffer much if there are longish delays in moving things about or constrained bandwidth. Real-time video conferencing across state lines requires fairly snappy response times and potentially lots of uninterrupted bandwidth. The value of different qualities of bandwidth differ. In order to maximize the economic efficiency of network investment, we need to be able to set different tariffs for bandwidth of differing value. Unfortunately, we cannot trust any of the incumbent network providers not to abuse such a thing and we cannot trust the system overall to protect the disadvantaged from being pushed out into a second class slow lane.
The ideal would be to have nothing but ultra-low latency and essentially unlimited, uninterrupted bandwidth. That is, the ideal would be if there was only one single quality of bandwidth that was adequate for all needs. That is not likely to happen on a real network for the foreseeable future and hence we need to be realistic about how we charge for different types of bandwidth.
There is much that requires improvement on our global network. I don't think that the status quo of ridiculous confusion is ultimately helping anybody. It certainly is not maximizing the greatest good.
If it's a Netflix server, then Netflix should pay for it. And for the traffic between Netflix servers, and for the power, and perhaps even for the floor space. Ideally the Netflix server would be on the other side of a demarcation barrier, and shove their Cat 6 through a glory hole to the ISP. The only requirement on the ISP should be to efficiently accept all the Cat 6 offered. Netflix pays for their side of the wall.
Even the Pr0n CDNs pay their own bills. So Netflix is a step down from pornographers. Trying to be Netflix without standing up a CDN.
It really sounds to me like Netflix have a very popular product and a model for delivering it to the customers. One may argue advantages, disadvantages, "fairness", "wastefulness" (caching/no caching/whatever), or other features and qualities of this model, but let's assume for the sake of this discussion that the model suits Netflix's current business needs. That is really all that matters.
The ISP's customers want that product. Delivering it to the customers costs the ISP extra compared to the rest of the content it carries. IMHO, the ISP has two choices: say, "we don't carry it" and hope not too many will care, or pay the price and pass it on to their customers one way or the other. It is not fundamentally different from a local (brick) store whose clientèle wants products from (say) an overseas manufacturer. It would also fight an uphill battle with (e.g.) major chains that have the infrastructure and relationships and economy of scale in place.
[The direct comparison with the brick-and-mortar world tells me the situation has little to do with "net neutrality".]
It is perfectly legitimate to complain about the situation, of course. Complaining, however, will not be a viable third choice *unless* a lot of people will forego Netflix "because it is unfair to small ISPs" and thus force Netflix into revising the distribution model, rather than switch to a different ISP that has the goods.
"Costs extra compared to the rest of the content".
No, it costs the same. If you send 10 gb of emails, it costs the same as 10gb of netflix in bandwidth and pipe use. The difference is before the customers did not use the ISP at all, they just sent emails.
Now the customer wishes to use the service they are paying to the full. This creates a problem, as the ISP was playing the "spread the costs over low users" game. Which leaves them in a sticky situation when the real figures show they cannot server the customers with the product they were charging for...
(This sometimes happens with airline companies. They sell 310 seats on a 300 seat plane in hope 10 people don't show up because of reasons, like work turning down holiday or being ill with a cold etc)
If there wasn't any DRM, services like Netflix would easily cache via a transparent cache and we wouldn't have that problem.
Other than that it seriously makes me wonder how bad the infrastructure in the US must be that ISPs actually cannot get proper affordable upstream bandwidth.
I have no love for DRM or for Hollywood. And I don't even care about watching content. As far I can see it's all crap anyway. But it's not true that DRM stops caching.
It's perfectly possible to cache data without breaking DRM. You merely need to make each block identifiable so the cache can tell when it's needed again. This doesn't mean you're caching the film in plaintext.
The point of the cache network-wise is so that more than one person can access the same content without having to download it twice. This falls apart in the Netflix model because EACH user has a different encryption key, so the same copy delivered to two different users looks different during transit: resulting in a cache miss. And the content owners insist on it being this way end-to-end. They don't want the content transiting in ANY kind of "common" format: even a common encrypted format because, at some point, the content has to be transcrypted to the user's key, and it's there that a man in the middle can pirate the content. So in the end, because of the content owners, caches are useless for the purpose of significantly reducing common traffic because, essentially, NONE of the traffic is common at all, as they ALL pass with different encryption keys.
This ISP simply wants to charge both ends just like Comcast. The fact is the customers want this content and are already paying the ISP for access. I wouldn't be surprised if there is some underhanded monetary transaction with Comcast as they'll stop at nothing to get their monopolistic way.
Given that Netflix tends to aggravate their upstream costs, which are ALWAYS metered, perhaps there's some measure of fairness in it. Even when it comes to shipping physical things, there's some give and take involved. Sometimes, the buyer pays the shipping; other times the supplier eats the costs. Perhaps the next question to ask is whether or not the amount the customer pays between the ISP and Netflix is sufficient to fund all the upstream costs. If it's not sufficient, then the ISP probably has a case to ask for compensation from either end. It's something that has to be hashed out between all parties involved, just as bulk shippers need to cut deals with transport companies.
Net neutrality != internal deals with ISIS
Netflix may well cause a lot of issues with lack of caching and connection deals but this is a separate issue from the much broader net neutrality debate.
Aspects of the Verison/Netflix row highlights how ISPs are already abusing the current rules and how this abuse can be extended and made legal by the new proposed rules.
Agreed. The argument from the ISP is that Netflix should stop putting so much effort into net-neut lobbying and more cash into caching. The two issues are unrelated except that the resources being put into one are eating up resources which really need to be put into the other.
While I understand the ISP's problems and irritations with Netflix, we do need a well-funded net-neut supporter. Sub-headline quotes were not well chosen as it appeared that the ISP was anti-net-neut (pay for traffic) when in fact they just want them to pay for their own CDN kit, which is quite reasonable and not anti-net-neut at all.
There is no little reason for the Netflix boxes not to run as a proxy. Even Squid can do some clever stuff with pulling content from youtube and then serving it as static content from a local web server. There's no reason for netflix not to do something similar and ease the load on systems where much of the data is never served. The performance hit from running as a proxy is nothing more than having no content servers at all and its limited to the first user. There's no reason not to pre-cache things you know will be popular. For the sake of a slight performance boost for the first user, Netflix is being rather obnoxious.
I'm a little surprised by the proxy/server ratio in the real world. Here in Australia we have really poor access to the Packman repo's. There are mirrors but the update lags cause problems with failed package installations. I have no idea why you would want to dedicate disk to a mirror when you could offer a front-end proxy which will always be up to date.
So basically what he's saying is he doesn't want to look like the bad guy to his customers. His customers are demanding better netflix performance. The appropriate thing in this situation to do, if a majority of his customers are asking for it, is to run the line to the pop where Netflix is at, and charge the customers more money to make up for the cost. Instead, he wants to whine that Netflix should come to him.
Two problems with that:
1. Netflix doesn't run an ISP. They don't run fiber, they aren't a service provider. If their current bandwidht providers don't service the pop this guy is in, that doesn't really appear to be their problem to me. His problem is with Cogent and Level3, not Netflix (but he already knows that)
2. What is their incentive to do so? So that he can continue charging his customers the same money while Netflix incurs additional cost?
No, it isn't Netflix' problem that you don't have a peering agreement with your uplinks at their nearest pop to you. It also isn't their problem you aren't a tier1 and have to pay for your transit. If your customers want it badly enough, they'll pay for it. Hell, send them all a survey saying "hey, the only way we can make netflix faster is to spend a bunch of money, here's what it's going to cost, would you like us to do it?".
I am the independent ISP mentioned in the article. Some of the comments above have been unkind or misleading; I'll try to correct these and add some more information that was not included in the article here. Note that we are a small, hard working local company undertaking a difficult task for our users despite attempts by corporate giants thousands of times our size to crush us, so please save the ISP-bashing for your monopoly telephone or cable company.
1. Netflix' "OpenConnect" appliances are not caches. They are hosted servers, which Netflix wants ISPs to host, for free, exclusively for Netflix. (So much for any pretense of "neutrality.") Every day, Netflix loads terabytes (that's not an exaggeration) of content into the server -- including multiple copies of every movie, in different formats -- and expects the ISP to provide the bandwidth for this for free, even though most of it will never be viewed. This saves Netflix money on CDN charges, but actually costs the ISP quite a bit. The $10,000/month bandwidth cost mentioned in the article is not an exaggeration; it is what backbone provider Level3 just quoted us for a "gig wave" (a gigabit of fiber optic bandwidth to an Internet hub).
2. The Netflix appliances are also power hungry and consume rack space, air conditioning, and other expensive resources which ISPs are in the business of providing for a fee. But Netflix wants to leverage its market power to obtain all of these things for free. Verizon and Comcast have said "no" and have gotten Netflix to pay a fair price for their services; small ISPs have gotten nowhere (not even the same amount per subscriber, even though rural ISPs have higher costs per subscriber). We think it's fair that small ISPs be paid at least the same amount per subscriber (after all, our Netflix subscribers ought to be as valuable to Netflix as Comcast's and certainly deserve the same consideration from them) to cover bandwidth, rack space, and/or equipment to improve Netflix performance.
2. Transparent caching of Netflix content would be easy to achieve without compromising the content creators' intellectual property. All that is necessary is for the content to be encrypted, with the key sent to the player via a separate secure channel. Then, the cache itself could not be used for piracy. The player, of course, still could -- but this would be true with or without caching.
3. To ask the ISP to cover all the costs of connecting to Netflix and enhancing its performance is (a) asking the ISP to favor one content provider over another; (b) asking non-Netflix customers to subsidize Netflix customers; and (c) asking the ISP to subsdize Netflix. None of these things are fair or reasonable.
Our ISP does provide the amount of bandwidth we promise between the user and the nearest Internet peering point. That is what users pay us to do and what we are obliged to do. We do it as best we can, and at the best price we can offer, despite the high cost of serving a rural area. Most of our customers say we do it better than the monopoly telephone and cable companies in our area (we offer demonstrably lower latency and jitter), and many folks in town have switched to us from them. But Netflix apparently has performance problems even on a high quality connection if it does not go directly to a Netflix server.
Netflix is asking us to give it preferential treatment, asking us to give it services for free, wasting expensive resources, and refusing to allow us to simply implement an efficient LRU cache for Netflix AND other content (something that would cost us a bit of money, but would at least be provider-neutral; it would thus be fair to ask all of our subscribers to pay for it).
We, on the other hand, are asking for only two things which we think are fair and reasonable. The first is that Netflix not squander bandwidth, which is costly here, by allowing efficient caching of its content in a way that does not compromise intellectual property. The second is that it provide the same subsidy for each of our Netflix users that is already providing to improve service to its customers on Comcast and Verizon. We won't pocket the money; we'll spend it to improve performance, and we're willing to sign a contract that stipulates this.
Netflix -- a billion dollar company -- is trying to leverage its market power against us. I think you'll agree that that this is not fair, reasonable, or in any way "neutral."
That's patently false. Improving your connection to whatever the nearest pop Netflix is located in would in no way be preferring them. Nobody is saying to dedicate a cross-connect to Netflix. They use Level3 and Cogent, there's no possible way that if you ran to whatever pop they're in, you wouldn't also be improving your connection to countless other providers. Basically every non-mega-internet-corp in the world has a blend that includes Level3 or Cogent in their mix and you know it.
"Basically every non-mega-internet-corp in the world has a blend that includes Level3 or Cogent in their mix and you know it"
That would be the stereotypical view that America == World? :-)
I know level3, HE, and others have global links, but you'll find in the UK at least they are at public peering points, and sometimes with the big ISPs but definitely NOT the small to medium ones!
@Brett Glass:
Well put. To the best of my knowledge, you are accurate and fair in your description of the situation and your estimation of problems/solutions seems practical and otherwise sound to me.
There are broad systemic issues that should be corrected in order to arrive at a truly 'best' solution. However, that is well out of scope for a small ISP at the leaf nodes of the network. Meantime, what you propose is sane and sensible to avoid a greater evil. The larger incumbents of all stripes, NetFlix included, have done us no favors. They have proven, thus far, to be poor custodians of network infrastructure. It seems to me that smaller ISPs are more likely to accord with broad public interest than the cartels that currently hold sway.
There is, to some extent, a fundamental tension between the public good and what corporate entities do. Corporate entities are, by their nature, 'evil robots'. They cannot reasonably take into account the public good. Directors and executives have a fiduciary responsibility to work to maximize the narrow financial interests of their community of shareholders. To the extent that those interests conflict with any other community and the community at large, they cannot self-police in the public interest because their duty lies elsewhere.
It is necessary that the representatives of the people ensure that the overall value of the public good is maximized. To the extent that this conflicts with corporate interests, rules and enforcement mechanisms must be put in place to make the playing field level. There must be artificial inducements via rewards and penalties mandated from outside by the government to induce corporations to act in a way that results in maximum net benefit for the body politic.
The current status quo is contrary to public interest. It has been unduly influenced by communications cartel members and large players like MS, Google, NetFlix, facebook, etc via lobbying and improper influence on standards bodies. This has been compounded by a state apparatus that demonstrates to me that our government has become a self-interested entity unto itself; no longer capable of acting in the public interest.
A proper long-term cure for things like our network issues starts with sunlight. You have done a good job of shining a little light on things. You also have a responsibility to advocate for your company, so I take what you say with a grain of salt. However, it is pretty clear to me that your interests coincide with mine much better than the major players. As far as I can tell, you have drawn a pretty accurate picture and if nothing else, I think that your honesty should be rewarded by giving more weight to your words.
Unfortunately, if it's genuinely true that 'one of the first questions asked by every customer who calls us is, "How well do you stream Netflix?"' then they're really not your customers, they're Netflix customers.
It may not be "fair" or "reasonable" if you believe consumers shouldn't be limited in choice to a handful of faceless megacorporations, but I think you'd be hard pressed to identify many mature consumer-facing business sector in which this is not the case (hairdressing and prostitution are the only two that come immediately to mind...).
A big thank you for joining the forum and debate.
Netflix may well be a sod to deal with as a business & using net neutrality as a smoke-screen, but this issue must be seen separate from the FCC's net neutrality debate which has far wider repercussions.
A separate debate would be needed for the less than neutral behaviour of the content providers.
Quick question: Are you pro two tier Internet?
This may be an unfair question given your position, however considering that these concerns can be wrangled out after the FCC'S conclusions this is at best badly timed.
"2. Transparent caching of Netflix content would be easy to achieve without compromising the content creators' intellectual property. All that is necessary is for the content to be encrypted, with the key sent to the player via a separate secure channel. Then, the cache itself could not be used for piracy. The player, of course, still could -- but this would be true with or without caching."
Hollywood does not believe this to be secure. Even if the content is encrypted with a common key, in order for the content to be transcypted to the subscriber's key, the key must be delivered to the machine at some point. The fear of Hollywood is that someone (an insider, a hacker) can intercept the key inside the machine at some point, because it has to be decrypted at that specific point, allowing for man-in-the-middle piracy. For them, nothing less than end-to-end encryption will suffice, because if the stuff is pirated at the source, the law can concentrate on Netflix, while if it happens at the receiving end, they know where to look as well. This becomes a lot harder with a man in the middle.
One of the first questions asked by every customer who calls us is, "How well do you stream Netflix?"
It sounds to me like one of the main reasons people are paying for broadband is to get Netflix. Are the ISPs paying Netflix for this huge attraction?
It seems unfair that Netflix should put in all this work to get people using broadband, and not get at least some of the broadband subscriptions...
The fair approach is some approximation to "meet me halfway" ... unfortunately, Brett's complaint is that halfway across the US is still a long way, so his ISP can't afford to pay its own half. (Netflix are already paying Level3 or Cogent to provide the link as far as each peering point, then expect the ISP to pay between the peering point and themselves - just like the BBC does in the UK, for that matter.)
With Comcast, the situation had been Netflix --- Level3 --- Comcast, with Netflix paying the first bit; because of the scale involved, it seems it ended up cheaper for Netflix to pay for a direct link cutting Level3 out of the picture. (A 10Gbit pipe from Netflix into Level3's backbone turns out to be quite similar to a 10Gbit pipe from Netflix into Comcast's backbone, after all.)
I don't have much sympathy for Brett's demands for Netflix to pay him. The job of an ISP is to transport traffic between its users and the appropriate peering point*s*; if you cheap out and use only a tiny local peering point, of course you will get lousy results. Peering versus transit is a tradeoff - arguably the core one to running an ISP: connect to a good major peering point like LINX, you can offload a lot of your traffic more cheaply than running it over your transit links. Or, at the two extremes, you can have no peering at all and use transit for everything (the micro-ISP approach, where you are really just reselling bandwidth from a bigger ISP) or you can peer everywhere with everyone and not pay transit fees to anyone (like Level3, Cogent and co do - the mega-ISP approach, where you are running your own vast global network).
Peer with them (yes, that costs *both* parties money: Netflix aren't getting their connection to that peering point for free any more than you are), pay a bigger ISP to do it for you (transit), or explain to your customers that you won't provide enough bandwidth. Just don't ask another ISP's customers to pay for the bandwidth you're selling to your own customers.