Re: robots.txt
"Up to the news site to decide if their content not being visible via big name search engines is a good idea in terms of click throughs / monetization."
Isn't that a bit like saying that the Highways Agency can come and help themselves to your stuff in your front room, unless you say you don't want a road to your house?
We need to be absolutely clear about this: Google can either 1) do what it wants, or 2) be a monopoly. It should not be allowed both, as that will lead to massive abuse of power. Like, for example, taking people's stuff and slapping ads on it, then threatening with wiping them off the (public) face of the net if they disagree.
Biting the hand that feeds IT
