When algorithms attack, does Google hear you scream?

One day in June 2006, search startup Foundem vanished from Google. Foundem is the developer of a "universal vertical search" technology, and currently offers comparison shopping across a range of consumer and travel categories by drilling down into vendor sites and returning details of actual flights and products. But on 26 June …

[Anonymous Coward]

universal vertical search

"Analysis One day in June 2006, search startup Foundem vanished from Google. Foundem is the developer of a "universal vertical search" technology, and currently offers comparison shopping across a range of consumer and travel categories by drilling down into vendor sites and returning details of actual flights and products .."

Foundem is a search company, like Google. Why would it's search results turn up top on Google. Does Google flight search results turn up tops on ms Bing ?

[Number6]

Vertical Search Sites

In general I'm actually on Google's side here. The number of times I've been looking for something and had to wade through pages of crap that just leads to other search pages because I couldn't tune my search well enough to omit them all as irrelevant is huge. If I'm searching for a named item, direct hits should always be higher than the comparison sites unless I add the word 'comparison' or 'review' to my search.

[amanfromMars 1]

An Alien Strategy BetaTesting All Manner of Operating Systems Virtually

"Google will know everything about everything - including you - so it will know what you want, even if you don't know you want it. Just give in - you know it makes sense."

Err.... Google will think they know everything about everything, and will be very easily groomed with what you want, if they want what you have and are feeding into the web for their algorithm/search engine/business.

Quite what the Google business is really all about, is one which you, and they, would not really like you to hear? And posed as a question because a statement would be just too harsh and infer that they have been rumbled/tumbled/outed/uncovered and unmasked/betrayed/whatever, and would need to Stealthily Invest in a Louisiana type Purchase for the Glorious Sound of Continuity Silence.

[N2]

@ Vertical Search Sites

Im with you here, Im fed up with wading through crappy price comparison sites.

but Foundem seems quite good & slick with it, so a potential competitor to Google itself

[Hayden Clark]

Grr - fake "review" sites!

What the shopping comparison sites have utterly broken is the ability to type "<product I'm thinking of buying> review" into the search box, and actually turn up reviews of the product. All you get are page after page of shopping sites, each with either "review" in the title, or containing a link that says "no review - click here to write one. Worse, try "<obsolete product on eBay> review" and you turn up dead pages from the same shopping sites!

[Peter Hartley]

Spamfestation of search results

Like poster Number6, I'm right behind any Google algorithm, special-cased or otherwise, that keeps price comparison sites out of the search results. I'd always assumed that Kelkoo, for instance, retained its annoying presence due to a bug in Google's algorithms rather than any deliberate "whitelisting". Google should provide a setting (implemented like the Safe Search setting) that lets the end-user turn the whitelisting off, so that Kelkoo, Foundem and the rest remain at 119th place or beyond, where they belong.

Peter

[Anonymous Coward]

Google - Minus the junk

http://www.givemebackmygoogle.com/

You might find this useful

[Anonymous Coward]

shopping sites

are totoal and utter rubbish. They are affilate crap, MFA rubbish. I wish google would delist them forever

[John Lettice]

Re: Spamfestation of search results

That would seem to me to be one of the problems. If price comparison sites are bad per se, then price comparison sites should be uniformly penalised.

But, Google itself does not think price comparison itself is bad. The introduction of price comparison under the banner of 'universal search' is, according to Google, in response to the Google equivalent of customer demand, and in this case I believe them. If you put in a search for a specific product, then you probably want that specific product, so in that case you're not likely to want a whole bunch of price comparison cruft shouting at you. But much more often, people will be running searches that imply they're shopping for stuff, in which case they'd surely welcome results that present them with relative costs. In which case price comparison sites are perhaps not bad per se.

So why are some penalised and others not? Neither The Reg nor the New York Times can get straight answers on this, and it seems to me that Google, simply, has no valid mechanism for assessing the value of price comparison sites. Except when it comes to its own, which are by definition perfect.

[John Lettice]

Re: shopping sites

Including its own? http://www.google.co.uk/products?hl=en

[Anonymous Coward]

Google's algorithms and AV heuristics...

have a lot in common.

I stopped using anti-virus when I realized that the rate of new viruses / techniques is simply too high for any AV company to keep up with.

I wonder if such a day will come for the web.

[MinionZero]

@ Vertical Search Sites

I also hate these Vertical Search sites. To me, they have always appeared like they were some kind of a scam, exploiting other companies keywords to focus Google onto their own site. So they are forcing themselves in between Google and the sites we want to visit. Which is probably how they earn some of their money from profiling what is clicked on. In effect, they are then like another form of spyware redirecting Google search clicks through their web sites.

[Robin 2]

The Perfect Answer requires the Perfect Question

"Eric Schmidt, for example, envisages Google eventually providing a single, perfect answer to your question: "We’ll get to the point – the long-term goal is to be able to give you one answer, which is exactly the right answer over time.... And what I’d like to do is to get to the point where we could read his site and then summarize what it says, and answer the question...""

Google and other search engines are already very good at providing an answer. The problem is knowing HOW to ask them for what you want....

Maybe there's a niche for some Stanford drop out: Build a question engine which generates the perfect question for Google to search with. A bit like Earth in H2G2...

[Trevor Pott o_O]

You Filthy Blasphemer!

If Google says something is bad, it's bad, mmmmkay?

If Google decides later that it's good, that's because it's NO LONGER BAD. That's all there is to it. Google just ISN'T EVIL. Microsoft is evil. Google is like Apple, it's just NOT EVIL. It cares about people, and it just does things right. They just work and aren't evil! How dare you question them?

I need a mocha chai latte with extra foam and double sprinkles. Or maybe a yerba mate. Oh I just can't decide...

See how your blasphemy is damaging my calm?

[Anonymous Coward]

I think google are right here

I would dearly love the option to select "no comparison or search sites". They are an absolute waste of space and time. I often click straight to page 2 of the search results to get rid of the crap that is served up before the real 'meat' is delivered.

[Ian Ferguson]

Google FAIL

Excellent article and investigation, thank you.

I have long felt that Google is shooting itself in the foot by operating a 'no categorisation' policy - insisting on one set of search results, no user options at all. This is great when the search results are clean and the search query is suitable; but in terms of comparison sites, it's Google's biggest weakness.

I would dearly love to see the option to show or hide comparison sites, along with review sites, shopping sites (sometimes I just want to go to the manufacturer's site, not see where I can buy an item), or indeed any kind of for-profit site (sometimes I just want independent opinion). The problem lies in the fact that Google do not want the task of categorisation or judgement - hence the 'it's the algorithm wot done it' argument.

At a Google-run SEO conference a number of years ago I put it to the speaker that business-led SEO is not always in the best interest of their users, and that shopping comparison sites are a prime example - she could not understand, insisting that if a business has the resources to optimise or money to advertise, they must be providing the most relevant content.

[Anonymous Coward]

We must perform a quirkafleeg...

Google needs to be regulated, plain and simple. As well as being more rigorously persued in terms of corporate monopolies / anti competative trading etc. There needs to be a regulator who can hold it (and indeed other search engines) to account as the sole providers of online search results / information.

Great in theory, but in practice very difficult; it would require an international 'ombudsmed' with enough clout to hold Google and others to account. However, without such a body the internet will increasingly become a very dark arena from a business ethics perspective, and potentialy from many other perspectives as information increasingly shapes and controls the world we live in.

Black helecopter in this case will be welcome, so long as it's flying over Google and aiming it's eyes / guns at them!

[Steve the Cynic]

Reviews...

I want to be able to search for reviews of stuff without seven 20-item pages of price comparison sites that don't actually have reviews on them. All I want is to read a review from a relevant site, e.g. Tom's Hardware or similar for PC parts, and I don't want to have to remember to type a series of -nameofsite in the search box. Where is the "no price comparison sites you f**kwits" checkbox on the search preferences?

[Trevor Pott o_O]

On a more serious note...

Excellent article! It serves to highlight some of the questionable areas of Google's online efforts. While I hesitate to apply deliberately malicious motives to the company as a whole, or even the senior staff, it would be completely exceptional for an organisation of Google's size to be immune to periodic bouts of incompetence or "left hand not knowing what the right foot is doing."

What from the outside can seem sinister plan is often merely unfortunate coincidence. One department is busily doing its thing, (say for example changing algorithms to boot out search sites that don’t produce their own content) and another arm of the company is busy expanding building universal search, or expanding on what froogle indexes. It’s not necessarily collusion to drive out competitors, but it certainly speaks of a lack of communication between all the various Googly tentacles.

The bit that really gets me about it all is the attitude. The apple-like culture of "we can do no wrong" that has sprung up around Google. Deny everything by default, not only officially and publicly, but internally as well. We can all understand and appreciate the need to deny things publicly to avoid legal liability, while behind the scenes running around finding out what went wrong.

When you truly and honestly believe that there can never possibly be anything wrong…that’s when innovation stops and hostility begins. Anyone who challenges the perfection of "the algorithm" or the "business model" is by definition a bad person or a crackpot. When the people responsible for support start to believe this, real issues with the code never get reported up the stack to developers who can actually fix bugs. When developers believe this, bugs never get fixed even if reported. When management believes this, money flows away from developer time to fix bugs and goes elsewhere.

Because of all of the above, as well as what is in your article and a few other trains of thought that are related, I believe Google’s core business has stalled out. They have stagnated on their search, and thus by extension the ability to refine their advertisement provisioning. (The adwords algorithms are probably not part of the search algorithms, but likely share a lot of commonalities. A bug in one will probably often be a bug in both. Errors in search are irrelevant to Google, however errors in adwords cost them money.)

While my fellow commenttards will no doubt rise up in numbers to both condemn and redeem Google according to their beliefs, there are two things I think we should all bear in mind: the algorithm as it stands today isn’t perfect. It is subject to spam and gaming and likes certain sites such as experts exchange and Wikipedia a little too much. The other thing is that there has been very little noticeable improvement in the past few years. (I realise this is a subjective assessment, however even minor searching finds all sorts of research to back this up.) Google has refined their algorithm continuously, trying to improve the signal-to-noise ratio, but they have done little more than keep their heads above water. With many search types I use regularly Google seems to have almost completely lost the battle; the noise almost completely overwhelms the signal.

Again, this doesn’t necessarily imply sinister design on behalf of our not evil overlords. (Note that is doesn’t rule sinister plans out, either.) It begs a whole lot of questions however…and Google has so far not proven big on providing anyone with answers.

[Owen Carter]

Example:

Google is basically doing the correct thing here for me as a consumer, but they are making a big mistake in allowing whitelisting or special consideration of -any- site in their index, they should focus on the algorithms and result relevence to consumers (and not result relevance for suppliers, search parasites, and media types).

If I search for:

'Carlos Fandango': I want the carlos fandoango homepage.

'Carlos Fandango sucks': I want commentary on the quality of Carlos Fandango

'Carlos Fandango Help': I want the manufacturers support site, or a good help forum

-None- of the above should ever return a shopping comparison site..ever. If such sites want to appear when I search these terms it should only be in the adverts, not in the search listings, and they can f***ing well pay for it.

Also if I search for:

Carlos Fandango Shop: I want sites that sell the products, not shopping comparison sites.

Basically there are a lot of parisitical sites, like Foundem, who's effect on the web, search and consumerism in general is long-term negative. The more proactive google is in setting it's algorithms so they favour direct consumer->supplier contact and reduce consumer->parasite->supplier results, the happier I will be.

I grudgingly admit that I'd have no problem with:

'Carlos Fandango Price Comparison': showing Foundem in the result list.

[Anonymous Coward]

Great Article!

Well done to all at the register, this kind of thought provoking, well researched investigation and journalism on important tech issues is why I read this site.

[pedrodude]

Nice one El Reg

I'm with commentards Trevor Pott and Steve 116. Excellent article!

Google should no doubt receive praise for it's swish, powerful, and usable product offerings but the extent of their hegemony throughout the internet means that even the most minor developments, such as the "hundreds" of changes to their search algorithm need to be thoroughly scrutinized as their effects are highly influential.

[Martin 6]

@universal vertical search

>Foundem is a search company, like Google.

If they could show Google blocked them because they were a competitor it would cost Google - like Microsoft's settlement for deliberately sending bad CSS to Opera browsers.

I'm sure Foundem could find somebody in Redmond willing to pay the lawyers for an anti-trust case against Google

[tiggertaebo]

Great article

<joke>Well know we know how Apple reviews apps for the app store - it uses a Google Algorithm</joke>

On a more serious note Google's inconsitencies are massive, while I'm not a fan of ANY price comaprison site (one of those things I think most of humanity would vote to un-invent) the way that some such as Foundem drop/get penalised yet many dozens of others, many of which are even worse when viewed using the Holy guidelines continue to clog our results. As others have said I find googling for product info to be worthless these days!

In terms of "when monopolies attack" I'd say that google is one of the worst - a serious drop in google rankings can almost destroy a business and that gives them absurd levels of power in the e-commerce arena.

PS: We need an evil google icon!

[Estariel]

Commission

Reason for prevalence of comparison sites = commission to Google for sales

Reason for whitelisting of certain comparision sites = bigger commission to Google for sales

Reason for blacklisting of certain comparision sites = lack of commisssion to Google = competition

Remember, you are NOT the customer for Google.....you are the product.

The advertisers are the customer.

[Bruce Ordway]

User defined blacklisting?

What I'd really like is to be able to blacklist sites myself.

There are some sites I never want to see in my search results.

i.e. Experts Exchange? NEVER....

I can block the domains with OpenDNS, but the links still appear in the search results page.

The first search engine that give me control will win me over. Until then, Google is the best there is.

[Foundem]

Google Penalty Myths

The secrecy surrounding Google’s new breed of ‘blameless’ penalties, coupled with the stigma attached to any Google penalty, has given rise to several common myths and misunderstandings that, in our view, suppress meaningful debate.

In an attempt to promote productive debate, we have written a summary and rebuttal of what we believe are some of the more common and recurring myths and misunderstandings. The document, “Google Penalty Myths”, can be found on our SearchNeutrality.org site.

Several of the comments above concern what we like to call myth no. 4, “Everybody hates price comparison sites” (though if the above comments are anything to go by, we should probably promote it to no.1). This is our response:

The Myth: It’s great that Google is blocking price comparison site X from its search results because I don’t like price comparison sites. These sites are always cluttering up my search results.

The Rebuttal: This argument generally arises from the false assumption that all searches are vague and open-ended. A search engine cannot read user’s minds to discern their intent when they search on something vague like “canon eos 500d”. So these days, search engines tend to take a pragmatic approach to such open-ended queries by returning a mix of results from a variety of different kinds of sites, such as price comparisons, review sites, forums, stores, and so on.

But many searches are not vague or open-ended. Proponents of the “everybody hates price comparison” argument tend to overlook the vast array of intent-specific queries that users routinely type into search engines, such as “compare prices canon eos 500d” or “best price canon eos 500d”. It clearly makes no sense to exclude price comparison sites from these results, yet, when a site is penalised, this is exactly what Google does. Once penalised, a site’s relevance to a query is effectively ignored for almost everything apart from its brand name.

Besides, even if you believe that all price comparison results are bad in all circumstances, it is clear that Google doesn’t agree with you. Google do not penalise anything like all price comparison sites. On the contrary, at the moment, well established brands (that would be conspicuous if suddenly absent) are effectively granted immunity, while emerging, and possibly highly innovative, brands are discriminated against. Most significantly (and worryingly), Google now routinely features its own price comparison results at or near the top of all product-related search results, no matter how vague or open-ended.

There is perhaps a certain irony in the fact that much of the animosity sometimes directed at price comparison sites may originate from a period, not so long ago, when Google’s ranking algorithms allowed certain price comparison sites to run roughshod over many of Google’s search results. In the UK, for example, it was not uncommon to find a certain European price comparison site utterly dominating the first three or four pages of Google’s search results for most product-related searches.

[Anonymous Coward]

Consider the problem with News and News Archives

There are similar issues surrounding what is considered important news and with what can be found in news archives. There are obvious factors outside Google that affect what becomes a big news story. For example, anything that runs on the AP news wire is reposted on thousands of sites and is almost certain to become "news".

But there are other specialized news and journal archives and search engines that are a large part of the "archive" part of the Google search. Many of these sites make no secret that they are filtering their results (supposedly to provide higher quality results). When searching on specialized items on Google, many of the search results are being directed to sites requiring fees to access their archives (e.g. Highbeam Research and its many aliases).

When the flow of information becomes highly concentrated in a few places, these places become extremely high valued targets for manipulation. We see that routinely as armies of blogbots and twitnoids push hijacked pages to the top of results for hot search topics, poisoning the Google cache, etc.

We already have Google and other sites returning different content based on who made the request, where they are located, and when the request was made. How far are we from rewriting old news stories and tossing the previous archived version into the Memory Hole?

[Anonymous Coward]

An unsolvable problem?

This story well illustrates the fundamental dichotomy associated with Google.

Raw, unfiltered search results are worthless. But as soon as any ranking or quality metric is applied, the results become suspect, particularly if the ranking process is not open and transparent. Since Google generates money from advertising and the delivery of sponsored results, it has a conflict of interest which is near to impossible to resolve.

There is no easy way out of this problem. If there is not going to be publication of the ranking policies (both algorithmic and human), the only possibility might be to have search ranking that was truly personal, based on the individual user's feedback to tune the system to deliver what they personally consider to be valuable. Whether such tuning could be done based on less than several year's of use, I don't know, but you could allow individually controlled aggregation where the user specifically identified other trusted individuals whose selections were incorporated with their own to deliver a ranking mechanism.

Of course, finding someone who is willing and able to deliver such a mechanism, free of the taint of advertising, is likely to be a fruitless search (pun intended.)

[Anonymous Coward]

No evil, really?

"Since the primary purpose of the site is to drive traffic to other websites, the Quality Team has decided that the initial evaluation was not in error."

.... what was the primary meaning of Google itself? As far as I've seen, there are not much anything but links in Google (main site).

I'd say that is an plain English admission that they won't tolerate competition and definetely won't index those and there's nothing wrong in that.

Instead millions of BS "comparison sites" or totally worthless "search engines" appear to results, they are not competition (because they are so useless). There's no way to exclude those either.

Google's "No evil" policies aren't very believable nowadays, these policies aren't accidents, but company policies.

[Anonymous Coward]

Reviews...

re: Steve the Cynic

> All I want is to read a review from a relevant site, e.g. Tom's Hardware or similar for PC parts, and I don't want to have to remember to type a series of -nameofsite in the search box

Why don't you just go directly to the reputable review sites and look there? Or use a review aggregator site?

[Anonymous Coward]

Re: shopping sites

At least i remember froogle.co.uk which I use all the time. It does not cause the same issues as these other vertical search sites.

If im looking for a review of something a serious review, I just cannot search google for "product number review" because all these sites and theres dozzens of them (because it can be extremly lucrative) have the word review on it, but no reviews.

I tend to broaden search for reviews instead go to youtube, amazon or some other extremly large site that has reviews.

The same grip exists for wikipedia/digg/slashdot ranking better for certain keywords than the original site dedicated to the topic did but this has changed in recent times and I have noticed that they do not constantly dominate the top spot.

So I have no sympathy for any vertical search, powered by affiliates/ads, crap site that floods googles index.

[Anonymous Coward]

Semantic Web

I think largely many of the problems google face are the space that the semantic web is supposed to fill. Certainty about its success is a cloudy.

The kind of questions many are asking here require a level of meaning and understanding about the data that is being indexed (ie this data is: a review, a product, a product in a comparator service, a set of products in a 3rd party aggregator). Keyword / Link Analysis cannot provide this sophistication except in a vague way.

While a crawler with indexing, ranking algos and "magic" like a stemmer have provided search with a leap forward beyond the four decade old constraints of relational SQL this approach too has its limitations particularly in complex use cases - the results unless qualifed by additional semantics remain fuzzy and innaccurate.

Until a time when web masters publish structured semantic declarations about their products that conform to universally recognised schemas (eg. this bundle of data represents : a flight booking, a car, a dvd player, a tv programme, theatre tickets) this will remain a limitation.

In the real world oweing to competitive interests this might never happen in an open way.

If i run a flight quoting system why would i want to publish my products via a sematic api allowing a set of 3rd party "clone" sites to re-publish my products and compete against me? Its a similar argument that the media rights owners are now levelling at google who they accuse of "stealing" their content.

[TeeCee]

Re: I think google are right here

I think that the fact that you find that the first page is always saturated with comparison sites causing you to click stright through to page 2 as a habit is rather more proof that Google are lying through their teeth here.

You may feel that they are right in their stated principle, but your own experience would indicate that their actual practice is some way removed from that principle.

"Do as I say, not as I do". Some might take that attitude as one commonly used by perpetrizers of eviltude.

[David 66]

grrr

I hate these sites. Screw them all. No offense.

[TeeCee]

@Steve 70

Quirkafleeg?

I missed the bit in the article that mentioned dead small furry animals. Are you hinting that Google also gas kittens?

[Anonymous Coward]

Russian Dolls - The Onion Routing Paradigm

This highlights the Russian Doll / Onion Routing Paradigm - you peel back layer after layer after layer in the hope of getting to the core...

Here its google aggregating the vertical search aggregators collecting products from white labelled branded outsourcers in turn promoting offerings from a conglomeration of brand owners in business to package and badge stuff from suppliers who provide catalogue and act as resellers from a range of vendors who partner with other vendors....

Is there actually a Herbert actually making / doing something or is it just mostly an economy based on hot air these days?

[Anonymous Coward]

Are we human?

"We’ll get to the point – the long-term goal is to be able to give you one answer, which is exactly the right answer over time.... "

Is Eric Schmidt really one sandwich short of a picnic?

Is he such an intellectual simpleton that he wants exactly one answer to any question you could think to ask regardless of the nature of the question? Sod the subtleties and nuances of human thought - Eric would like just one answer for every question, thanks.

"Is there a God"? Yes or no, yes or no, yes or no - come on ... hurry up ... place your bets ... yes or no?

Betting ends.

No, there is no God and that's the only answer Eric ever wants.

"Are we human or are we dancer"?

Human or Dancer - what's it to be?

Well?

"My foot hurts - what's wrong with it"?

Gangrene.

Always gangrene.

Either this article misrepresents Eric Schmidt or he should be changing his name to Eric Schmuck.

[Anonymous Coward]

my results

re: Google algorithms # By Anonymous Coward Posted Thursday 19th November 2009 16:11 GMT

for a while i was adding -site:experts-exchange.com to every query, unti one day when i left out the '-' and found out about gender modification.

Google isn't the only one with this problem, one of my equipment vendors' searches appears to add 'VPN' to every search. It shows up highlighted along with the search terms that I entered...

I followed one of the links on givemebackmygoogle and found where you can modify the search string, so I'll be testing that come monday.

BTW, if your site continues to get dimished results, and you're not embarassed about it, you should have written something like "my site www.uniquewidgets.co.uk doesn't get good results even though we are one of a very few (tens) of companies that sell them online." but hell, I don't sell widgets.