Yahoo! looks beyond Google's data cruncher

Nowadays, when it comes to crunching epic amounts of web data, Google's MapReduce credo is all the rage. The Mountain View method of distributing back-end compute tasks across a sea of commodity machines has given rise to the open source Hadoop platform, which now underpins Yahoo!, Facebook, and even a chunk of Microsoft Bing …

[Bob 18]

Back to the Future...

The idea that MapReduce might not be the be-all and end-all of parallel computing shouldn't be surprising. Consider the more "traditional" field of large-scale scientific computing (i.e. solving PDEs). MapReduce is absolutely NOT the fundamental way to get parallelism because the different parallel parts need to communicate on a fine-grained scale.

The surprise to me was that something as simple as MapReduce was able to solve AS MANY problems as it has.

[Mr Young]

I'm not sure?

So does this mean Yahoo! considers my web connection a distributed resource? Maybe I should send them I bill for using my leccy and PC? Hey - what about the cost of lighting, heating, beer, CPU cooling,coffees,biscuits? OMG - spreadsheet required...

[DZ-Jay]

Re: Back to the Future...

@Bob 18:

That shouldn't really be surprising. As they say, when all you have is a hammer, everything looks like a nail.

Ever since MapReduce was described in that seminal white-paper, it became all the rage, and a lot of people did see it as the end-all, be all or distributed processing; and I posit that *because* of that, a lot of people have replaced their entire toolbox with this single hammer.

So then, it's not that MapReduce is able to solve so many problems, it's that it's been applied to so many problems, irrespective of its disposition to solve such problems.

-dZ.