Why you subsidize Google's Soviet-style Net

Google certainly likes to throw a party. Its annual, roving Zeitgeist conference sees politicians and policy-makers beating a path to the door for inspiration. As a result, Google and policy makers have got very, very cosy together. But how much are ordinary net users and taxpayers subsidising Google? The answer is a lot, and …

[Jim Preis]

It doesn't end there...

Although Google didn't leave the starting blocks until 1997, I'm pretty sure they were behind the staged lunar landings in '69 as well.

Pretty sure that my hair loss is also attributable to SupaG, but I need to build the rationale for that. WIll forward when complete.

Keep up the good work - Maroons!

Love,

Jim

[Paul 5]

what's the problem?

So you pay for the bandwidth you use to download google videos, and if you have a website, a minute proportion of your site's monthoy traffic allowance is used by google to look at your site so it can tell other interested users who might be searching for that sort of thing -- and that's somehow a bad thing? Or a subsidy someone is paying google?

I wonder who else could possibly be collecting such "subsidies"....

Do you want Google to pay for BOTH ends of every connection to any of their servers?

[Andrew Orlowski]

Re: what's the problem?

"Or a subsidy someone is paying google?"

In effect, yes. Every "producer" in the world is subsidising the maintenance of the Google search index. We don't hear too many complaints today, because it's generally considered to be a price worth paying today. Nobody wants to drop out of Google, although newspapers are very tempted to do so, if they could believe they could all do so at once.

"Do you want Google to pay for BOTH ends of every connection to any of their servers?"

It's not my call, it's up to people who pay the bills.

I'll describe it another way: the spread of costs as they are now means Google receiving a subsidy from producers. If this doesn't match the perceived value, then a new settlement will be reached, one with a different spread of costs.

Maybe Google will start to pay the producers who create the value (amateur or professional), or the distributors who ship it from the POP. The BBC was talking about creating a CDN to do just that - to help out the ISPs. Who knows?

[amanfromMars 1]

If you Like ....

"Maybe Google will start to pay the producers who create the value (amateur or professional),..."

Who will Spend what they are Paid and Rebuild the Economic Engine with AI Creative Spend Economy Drive System ......... an Alien InfraStructure Model with Always Available Flash Cash/Instant Delivered Credit for Virtuous BIGCHEESE Spends...... for Presents Virtually Delivered with FutureBuilders Anonymous.

For Reality ..... or Virtual Reality, if you Prefer.

To Work, Rest and Play in Both is an XSSXXXXPerience . Are you XSSXXXXPerienced and Loving IT in Live Operational Virtual Environments ....... Seventh Heavens, Way beyond the 42nd Degree of Enlightenment.

[Disco-Legend-Zeke]

Google wants...

...to make the internet back into the Internet.

Traffic is money. Shared pipes lower the cost of generating traffic, and the Googers just need to increase traffic to make more of it. Until you can download or stream HD movies, Google won't be able to sell ads in them or in search results pointing to them.

How many made for Internet movies are there already? If you wanted a video of the disco legend, where would you search for it? Google.

They want not only to share that pipe with other ISP types, but also provide front end services for the producers.

Once Google is in your house, they will also have local content storage in your water heater and your set top box. You might even want the optional picocell on your roof. Moore's law has been working on com stuff too.

For myself as a content producer, Google has been my only reliable road to monetization. As an ISP, I , for one, look forward to our new omnsient overlords.

[Paul 5]

That really would be a subsidy

>> "Do you want Google to pay for BOTH ends of every

>> connection to any of their servers?"

> It's not my call, it's up to people who pay the bills.

> I'll describe it another way: the spread of costs as they are now

> means Google receiving a subsidy from producers.

It's not really "up to the people who pay the bills".

If all users are going to pay for both ends of all connections (or is your complaint uniquely directed at Google?), then the ISPs will be double-charging everyone, and there will be a very real subsidy - by all Internet users, to the networks.

Who wants that sort of world? Apart from the networks, obviously...

The charging model on which the internet is based is that users of any type pay for the cost of their connection to "the Internet" (incoming and outgoing traffic), and once connected, they can connect to any other machine which is part of the Internet. Also that network operators choose which other network operators they exchange data with, in order to get the necessary connectivity to the whole Internet.

You seem to be proposing a new mechanism. It's up to you to show exactly how that would work on the real-world Internet, and to show that it is more efficient, and fairer, than the current scheme. You have done none of these.

[Lou Gosselin]

@what's the problem

Consider two content distribution solutions, P2P versus centralized. With one, all traffic between ISP customers and a web host must travel over an expensive WAN for each consumer. With the other, ISP customers can exchange bulk files/streams locally with the WAN only needed to refresh stale content occasionally.

The peer to peer technologies developed in the past decade are faster, more robust, more efficient, and cheaper to operate than centralized servers. P2P can even use DRM to protect copyright as (in)effectively as centralized servers, so that's not a valid excuse to avoid P2P for content distribution.

Unfortunately google's business model of monitoring keywords and injecting customized ads into other people's content preclude it from effectively adopting the obviously better, more efficient P2P technology in bandwidth hungry applications like youtube. Viable P2P has existed in many forms since the days of napster without any commercial technical support whatsoever, so it's quite obvious the impediments are political rather than technical.

Now I wouldn't say google is alone in it's desire to build centralized services which they can control, but it is never the less guilty of pushing those centralized data centers in favor of P2P.

So, in a very real sense, depending on the proportion of bandwidth used by youtube, google's business model can be said to directly contribute to inefficient utilization of WANs. That's something we all end up paying for whether we want to or not.

This doesn't even touch on the argument that google consumes more bandwidth than it pays for as hinted in the article.

[Anonymous Coward]

Normal business

Sounds perfectly normal to me - most companies demand corporate welfare, the more you hear about "market forces" and all that bullshit, the deeper their noses will be in the trough.

[Rob Aley]

car analogy

When I drive to the supermarket, it costs me money in petrol, insurance, road tax etc. However you wouldn't claim I'm subsidising the supermarket. The bandwidth I pay for to access google services (and promote my websites on their search engine) I would argue is in the same class.

[Curt Vile]

I've paid Google my share...

I used to admin a website with a large and sprawling forum, and Google's bot insisted on blithely ignoring the robots.txt and crawling every damn forum thread page by sodding page, to the tune of 2GB or so of bandwidth every couple of months. I eventually had to block it in the .htaccess from everything but the site's index page. I also took the opportunity to block every other damn bot and search spider, but that's another story...

[Disco-Legend-Zeke]

Two Gigabytes...

...is 20 cents (us) worth of bandwidth.

One or two clicks on adsense should recoup that nicely.

[Jamie Jones]

2GB?

My quite modest server package has a 4,000GB transfer limit per month, and I only average 300GB.

Of course, you could argue that the extra google traffic requires more equipment at the hosting company, and therefore I pay indirectly, but 2GB is nothing these days...

[Anonymous Coward]

Please do the math.

40 Megabit Metro ethernet is $2000 for a business. Thats a private pipe to the Internet brand peering point. This it the smallest amount of Real Internet(tm) you can buy.

So we just convert bits to bytes. Multiply by seconds are in a month? Remember it is bidirectional so double. I can punch it into the calculator, but can't figure out where the decimal point goes. And what about ethernet overhead.?

Most Thanks for your help.

[frank ly]

Google Bots Bad Behaviour Binge Burns Bandwidth

"...Google's bot insisted on blithely ignoring the robots.txt and crawling every damn forum thread page by sodding page.."

I have read, many times, that Google's bots are very well behaved when it comes to robots.txt compliance. Have I been misled or did you not set it up properly?

[Harry]

"you're paying for inclusion in Google"

That's stretching the imagination more than just a little.

Most webmasters are happy that Google accesses their content, because without it they would get far fewer visits.

Any webmaster that doesn't want Google accessing their content can set up the appropriate entries in robots.txt and Google will not bother them again. Nor will Google's customers.

Very few webmasters exclude Google, though almost all webmasters know they could if they wanted. That speaks enough by itself.

[Orjan]

Apples and oranges

Your YouTube streams aren't Google's bandwidth usage. You must separate the traffic Google creates for its own use (i.e. indexing bots, page and image caching, data replication), and what other net users creates by accessing Google properties (YouTube, GMail etc).

[Anonymous Coward]

Agree

I don't find the comparison entirely fair either, but I find it to be very typical for telco-argumentation. I already pay for the bandwith I use for watching Youtube-stuff, Google shouldn't be blamed for my intellectual shortcomings. I'm in several minds about the bots.

On one hand, they take bandwith which costs real money. Not a lot, but it's a valid point.

On the other hand, they generate user traffic which is presumably what the intarwebs is all about. There may be some websites that already have all the users they want thankyouverymuch, but I suspect that most would welcome just a tiny smidgeon of interest once in a while. So bots can be said to be multipliers in that the bandwith they cost comes back when users start popping in for a look.

On the gripping hand, Google bots can be a source of income if one can stomach the abomination that is Adsense. Maybe even enough to offset the bytes they consume. Dunno. Not enough data.

Oh, and I do believe that Google pays for the fibres running into their facilities (probably - in a just world they would).

I think you've been had by a telco lobbyist, Andrew. ;-)

WTF because all is not what it seems to be.

[burundi]

Don't agree

This article is making some tenuous leaps imo, I think the point of the current architecture is that it's cheaper to run a cable from point A to point B than it is to build data provisioning and caching at point A and point B.

Also consumers have a choice of paying lower prices for limited bandwidth per month or a higher price for unlimited bandwidth, it's the consumers choice to use/abuse this bandwidth so to see it as a subsidy for any particular content provider is disingenuous.

I'm happy to pay a premium for a service provider that will get the content I want to me quickly and without limitations, others may not be and so shouldn't expect to be able to stream HD content from the such as the BBC or Youtube at peak times without issues.

[Phil Bennett]

Customer bandwidth shouldn't be charged to Google

Am I missing something? The $344 million is Google's bandwidth bill, the $44 billion is all the customers' bandwidth bills. Google apparently gets 16.5% of all bandwidth usage in the US (Really? That's a Tesco-like dominance. Worrying) so that'd be $7.26 billion's worth of customer bills. That isn't a subsidy.

Well, you could argue that Google is providing, at a cost to themselves of $344 million, $7.26 billion worth of business for ISPs - after all, if customers used 16.5% less bandwidth, they'd want to pay 16.5% less right? So its a subsidy for the ISPs. You seem to be saying Google should pay customer's bandwidth bills for the priviledge of delivering them videos that they asked for - are you saying shops should pay for customer's cars?

Oh, and as for the centralised net thing, CDN is almost certainly the way to go - can you imagine the size the cabinets would have to be? And thats assuming the neighbourhood scrotes haven't ripped it open and nicked everything. Oh, and updates to every cabinet in the world whenever something new came out would be horribly wasteful when most stuff isn't going to be watched by that many people.

[Anonymous Coward]

You are missing the point

Andrew, when are you going to actually get it? I am posting this anonymously as this is actually my exact day job in a major telco.

Google architecture is incredible inefficient in a world where you _CAN_ cache content at every corner. That however is not an option. That was the Internet in the late 1990-es when it was ISP driven and predominantly IP based. It is not now. The last 10 years the telecommunications companies spent buying ISPs and redoing them according to their model where they _DRAG_ all of their traffic to a single (or small set of) location(s). UK is a prime example with most of the UK Internet contained in a single building on Coriander Avenue in London. In a world like that Google is _AT_ _LEAST_ as efficient as any telco.

From there on the cost of disaster recovery takes care of things. Google is load balanced across the globe with at least N:N-1 redundancy. Telcos are usually 1:1 redundant in everything service included. As a result Google is guaranteed to be _MORE_ cost efficient in today's world.

As far as us subsidising Google, well if telcos have an objection to Google traffic they can de-peer it. Anyone with the guts to do so? Anyone? Raise a hand anyone? Nobody... OK in that case there is nothing more to be said here.

[Andrew Orlowski]

Re: You are missing the point

"OK in that case there is nothing more to be said here."

Not really. Things evolve, and it's the future shape of the internet we're looking at here.

[James Hughes 1]

Street cabinets

have a major project - robustness. If the caching system on a cabinet fails (and now you are putting complexity in, they will fail more often), the entire street fails. If a Google (or AN Other) server fails there is an instant fallback to a working one.

Once street cabinets get this sort of failsafe level, then they become too expensive to install and maintain. To be honest, to cache all that is required in the cabinet to do what is suggested in the article would require many terabytes, probably peta or exa bytes of storage. Not likely to happen in the near future for cost reasons. And of course, you need to get the data there in the first place, which requires lots of bandwidth. Hey, this idea keeps getting worse.

With regard to Google using up all our bandwidth - isn't a lot of their traffic handled over dark fibre, which they pay for themselves? Has this been taken in to account in the calculations in the article?

[Goat Jam]

Just a small correction

Dark fibre = fibre that is not connected up. The "dark" refers to the fact that no light is sent down the fibre, so if google are sending packets down unlit fibres then they are cleverer than I thought they were.

[James Hughes 1]

To quote from Wikipedia (I know...)

"The term was originally used when referring to the potential network capacity of telecommunication infrastructure, but now also refers to the increasingly common practice of leasing fibre optic cables from a network service provider."

[Frank Thomas]

just propaganda

so the author appears to be quite anti-network neutrality. that tells me he/she wants us to bow before our corporate overlords.

SBC Yahoo! DSL service, now with Google access (additional monthly charges may apply)...

[Richard IV]

just more propaganda

That's not the only model of a non-neutral network. As it happens, I agree that the one you outline in believing the doomsday model your Googly corporate overlords would have you swallow is not conducive to happiness on the tubes. There are others, however, such as separating/prioritising on service _type_ rather than destination. DSL, now with faster HD video access as an option (additional monthly charges may apply)...

Personally, I'd pay extra for a service that de-prioritises adverts if I thought for more than one second that the content creators got to see any of that cash rather than the ISP :)

[Not Fred31]

damn those freeloaders

ISPs must be livid that people are being given more and more compelling reasons to sign up because of Google. Dammit, life was simpler and happier before that outerweb came along.

Perhaps the author of this article might be better writing for luddites_unregistered.co.uk instead?

[Anonymous Coward]

Do you work for Google?

OK, name me one great movie Google has made? Or one decent label that it owns. Or one decent newspaper.

Google doesn't give me any reasons to sign up to the internet. Lately Google has been giving me reasons to go off-grid, with ideas like Google Buzz, and Schmidt saying I shouldn't do anything on the internet I'd want anybody to see.

Do you work for Google? It sure sounds like it.

[James Hughes 1]

harsh

Things you might like to sign up to the internet for ...

Google Search

Google Maps

Google Docs

Google Mail

Google Earth

And lots of other stuff not provided, for free, by Google.

Google have never claimed to make great movies, music, newspapers. Which is probably a good job for the suppliers of those services. What they provide is internet based services which are very popular, because they work and work well.

Not a google employee, just a google user.

[John 62]

sounds like the airline industry

the airline industry has hubs and superhubs (e.g. EWR, JFK, LGA or LHR, CDG, AMS), but I think there is room for CDN caches in street cabinets as well as the monster data centres. After all, there are the little landing strips Ryanair uses.

Government should just stop giving any business public money as a subsidy, or 'incentive'. It's just bribery out in the open.

[Mad Hacker]

You lost me at...

"...1Gbps fibre network - far faster than anyone needs today."

Actually I need even faster then that.

[Brent Beach]

Dumb pipes want to be paid for doing nothing

The telcos have yet to add value so they want to be paid for what Google does.

Absent google the net would have far less traffic than it does. Absent google people would not be able to find anything on the net. They would be using email and perhaps a few newspaper sites. The telcos would be providing 56K dialup to most of the world because most of the world would find nothing of interest on the net.

Instead of trying to leech off the google money tree, telcos should starting trying to innovate on their own. Absent telco innovation, google will walk around them to the customer within 10 years and they will be getting nothing but their extortionate cell phone charges.

Then google will start after that.

[Andrew Orlowski]

Re: Dumb pipes want to be paid for doing nothing

Your point about the telcos failure to innovate is well made, but when you assert:

"Absent google the net would have far less traffic than it does. Absent google people would not be able to find anything on the net."

then you're in trouble.

Without Google, DNS would get cleverer. People would type in Fox News, the New York Times or The Register and find what they're looking for. That's an intelligent network.

You should talk to Google. They know they're indexing a lot of crap, it's expensive, and nobody needs it.

The test would obviously be to pit Smarter DNS vs Google in a price war, and see who won.

[unitron]

Without Google? No, thank you too very much.

"Without Google, DNS would get cleverer. People would type in Fox News, the New York Times or The Register and find what they're looking for."

And with Google I find out which one of those three sites has what I'm looking for (or I find out that none of them do), and I save myself having to waste resources loading and opening some assault on my eyeballs and having to wade through that.

And unlike all the search options I tried before the advent of Google, I don't have to wade through page after page of attempts to get me to spend money on something when I was looking for information, and not necessarily purchasing opportunities.

Although I will admit that when I am actually trying to find something which I wish to buy even Google seems to want to show me other stuff instead.

[Anonymous Coward]

Peering and transit => cheap. Last mile => expensive

The content distribution networks sit logically in the ISPs, but all they do is reduce the traffic over the cheap bit of the network. In the UK at least, the bottleneck, the expensive bit, is the interconnect between the ISP and the end user, in part the so called "last mile", but also the BT-owned interconnect between the ISP and the network of BT exchanges which is used by smaller ISPs from AAISP to Zen. No amount of caching in today's architectures is going to reduce those costs. Is it different in the rest of the world?

Also, does anyone have any reliable figures for how much traffic out there is, shall we say, "dodgy"? I would not be at all surprised if the combination of newsgroup binaries, P2P, rapidshare etc, and "adult" streaming (paid for or not) matched or exceeded the traffic which Google generates and delivers. It surely does for half the folks I know at work (so-called "managers" included). Perhaps Andrew will be addressing that sector of the market too? (We just had the CES, isn't that the one that used to have an "adult" show alongside? Or am I misremembering?)

[Jolyon Ralph]

@Paul 5 - "Minute proportion"

Far from being a minute proportion, looking at my site's stats for Jan 2010

Viewed traffic: 466.82Gb

Google bot traffic: 55Gb

So that's more than 10% of the traffic from "real" visitors. Hardly "minute".

Still, I disagree with the logic of the article. We pay for internet access and we choose what we want to view and we're choosing Google. And as a webmaster, I'm paying for my bandwidth with the full knowledge that part of this is for Google to spider my site.

Don't penalise google just because they're popular. The system works, leave it alone please.

[Adam Azarchs]

What's the problem here?

Google provides a service "free" to customers. End-users running searches and watching YouTube videos are customers. Websites wanting to be indexed by Google also count as customers. Yes, being a customer means paying for the bandwidth. This doesn't mean you're subsidizing Google. If anything, it's the other way around - they pay for their own bandwidth, which they use to serve you, the customer, free of charge (from them). Networks might complain that their customers use a lot of bandwidth because of Google, but it's not a legitimate complaint at all - if Google wasn't doing what it did, their customers wouldn't be willing to pay as much for access to a less-useful internet.

Calling those datacenters inefficient because they're located far away is also more than a little disingenuous. While there are costs to distance, they are fairly small compared to the economies of scale that large datacenters get. Otherwise google wouldn't do it. Google owns and operates a pretty significant private backbone network, after all.

If I set up a tourist attraction on an island with a toll bridge you owned, would I be asked to pay a share of the toll for every person using that bridge to come see what I'm showing? Don't be ridiculous. You should be thanking me for getting you more customers. If all that extra traffic generates more costs than it brings in with more tolls, your tolls need to be higher - simple as that.

Networks are trying to have it both ways. They want to charge their customers for the privilege of accessing google, and then they want to charge google for encouraging so many demands from their customers. If networks don't have enough bandwidth availible, we should all just go back to paying per gigabyte of bandwidth. The only reason the network cares where that bandwidth is coming from is because it's coming from someone they realize has deep pockets. Not a good excuse.

[ChrisTMarsden]

Soviet-style? But there ARE no goodies

Its always better to allow comments on a story - and good that AndrewO is replying. I still scratch my tiny lawyer's head and don't see how a great headline really meets the story - or that Scott Cleland's stat is really a good starting point...but in any case I will ask Those That Know to help me understand.

I certainly quite agree that there are Roundheads and Cavaliers but no real goodies or necessarily baddies (though Vodafone's Sicilian newspaper distributor is trying) - but I digress for 300 pages here: Marsden (2010) Net Neutrality: Towards a Co-Regulatory Solution http://bit.ly/buQqi7 (PDF)

[dephormation.org.uk]

If webmasters were concerned they

would lockdown their web site config.

If there is a bandwidth problem at all, it is not Google. It is companies that leave their web site wide open for anyone (including Google) to access. For them the solution is entirely in their own hands. Lock down your web site, block Google: bandwidth consumption goes away for ever. Time to complete this task: 30 seconds.

Which is why the kebab wraps and media industry bleating about Google is such a hollow envious complaint.

A more serious problem are the hordes of talentless and unimaginative people in the telecoms/marketing industry who believe the world owes them a living.

[dephormation.org.uk]

Concerning Taxing Google...

... as Father Jack might have said, dat would be an ecumenical matter.

[Paul 5]

minute proportion

@ Rolyon

The google costs would be a minute proportion of any site's costs, for sites with a significant bandwidth requirement (and bill). For sites whose monthly traffic is comparable to the level of search engine robot traffic they receive, their bandwidth bill should be very small (if it isn't, you're on the wrong tariff). And if the spidering is taking up too much traffic, there is robots.txt to keep them out.

But we agree the author has things backwards (from any perspective but that of the telcos, perhaps).

[disgruntled yank]

Hmm

The Soviet-style network would be a Gooleg Archipelago?

[Jeff 11]

DNS isn't search

"Without Google, DNS would get cleverer. People would type in Fox News, the New York Times or The Register and find what they're looking for. That's an intelligent network."

Except that's a directory, not search. And even if it was, search algorithms and indexes need to be co-ordinated, updated and maintained centrally to work efficiently and reliably. And that's the point where one authority controls the search results. Sound familiar?

[Andrew Orlowski]

Re: DNS isn't search

You're right, but my point is that for some people some of the time, that's all they need. It might even be many people much of the time.

Think about the kind of queries Google receives, and what might satisfy its users at a lower cost than indexing the Entire World Wide Web of Spam.

[MyHeadIsSpinning]

We're Subsidising Google Bot-Pirates?

FTA: "Google’s search bots regularly copy every page on the Internet, some as frequently as every few seconds"

Google is allowed to copy the internets in their entirety, and get that content subsidised.

If we do this, we get called immoral and disgusting pirates and sued for millions.

Talk about double standards.

[Tom 13]

Typical leftist claptrap

Artificially define costs and prices, then pit the masses against the so-called pigopolist.

I have plenty of problems with Google, and mixed feeling about net Neutrality, but this is utter nonsense.

[Anonymous Coward]

The Register's usual rubbish about the EU

"Although Google now gains more in revenue than ITV, it pays no UK tax. It's entitled to do so under EU Law, so you'll have to blame the single market for that, not Google."

The Register hacks should remember that the UK benefits from the single market too (and perhaps more so than most countries). Think about all the UK companies which export goods and services to the European markets yet pay tax revenues to HMRC.

[ForthIsNotDead]

Can I be the first to say

Thank you Mr. Orlowski for enabling comments on your article, and actually responding to the comments.

Nice to have you here!

[Colin Tree]

Who benefits pays

If Google makes it's money through adds, Google should pay for the bandwidth.

If you want to use content which you find valuable, you pay for bandwidth.

A mixture of adds and content is a balanced payment.

Pay more for add free content, or ask for extra advertising and receive cheaper content.

You do a search through Google but all the first few pages of results are adds or diversions from the content you are seeking, Google pays.

(which is indirectly paid by the owners of the pages who paid extra to have the pages come up early in searches).

You get the answer you want up top, you pay.