hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Cosmin Lehene <cleh...@adobe.com>
Subject Re: map/reduce locality
Date Wed, 26 Nov 2008 17:51:30 GMT
Hi Yair,

I missed this. Didn't know about the TableInputFormat fix. Is this a 0.19 feature?

Thanks,
Cosmin


On 11/26/08 6:05 PM, "Yair Even-Zohar" <yaire@revenuescience.com> wrote:

You may also check the patch: TableInputFormatFix extends
TableInputFormat

The way they put it:

//
// Attempt to fix the localized nature of table segments.
// Compute table splits so that they are processed locally.
// Combine multiple splits to avoid the number of splits exceeding
numSplits.
// Sort the resulting splits so that the shortest ones are processed
last.
// The resulting savings in network bandwidth are significant (we
measured 60%).
//


Thanks
-Yair

-----Original Message-----
From: Cosmin Lehene [mailto:clehene@adobe.com]
Sent: Wednesday, November 26, 2008 5:36 AM
To: hbase-user@hadoop.apache.org
Subject: Re: map/reduce locality

It doesn't currently do that. However this seems to be on HBase roadmap.
See Data-Locality Awareness

The Hadoop map reduce framework does makes a best effort at running
tasks on the server hosting the task data after the dictum that its
cheaper moving the processing to the data rather than the inverse. HBase
needs smarts to assign regions to the region server that is running on
the server hosting the regions' data. HBase needs to supply map reduce
hints such that the Hadoop framework runs tasks beside the region server
hosting the task input. These changes will make for savings in network
I/O.

http://wiki.apache.org/hadoop/HBase/RoadMaps

Regards,
Cosmin

On 11/26/08 1:32 PM, "David Faitelson" <david@proactivemodeling.com>
wrote:

Hi,

Does HBase/Hadoop create map tasks on the same data node that
contains the region for the map task?

I know that Bigtable does something like that but I could not find
any mention of this optimization in the documentation of HBase.

Thanks,
David



Mime
View raw message