hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Purtell <apurt...@apache.org>
Subject Re: Dealing with data locality in the HBase Java API
Date Wed, 04 Mar 2015 18:32:05 GMT
Spark supports creating RDDs using Hadoop input and output formats (
. You can use our TableInputFormat (
or TableOutputFormat (
These divide work up according to the contours of the keyspace and provide
information to the framework on how to optimally place tasks on the cluster
for data locality. You may not need to do anything special. InputFormats
like TableInputFormat hand over an array of InputSplit (
to the framework so it can optimize task placement. Hadoop MapReduce takes
advantage of this information. I looked at Spark's HadoopRDD implementation
and it appears to make use of this information when partitioning the RDD.

You might also want to take a look at Ted Malaka's SparkOnHBase:

On Tue, Mar 3, 2015 at 9:46 PM, Gokul Balakrishnan <royalgok@gmail.com>

> Hello,
> I'm fairly new to HBase so would be grateful for any assistance.
> My project is as follows: use HBase as an underlying data store for an
> analytics cluster (powered by Apache Spark).
> In doing this, I'm wondering how I may set about leveraging the locality of
> the HBase data during processing (in other words, if the Spark instance is
> running on a node that also houses HBase data, how to make use of the local
> data first).
> Is there some form of metadata offered by the Java API which I could then
> use to organise the data into (virtual) groups based on the locality to be
> passed forward to Spark? It could be something that *identifies on which
> node a particular row resides*. I found [1] but I'm not sure if this is
> what I'm looking for. Could someone please point me in the right direction?
> [1] https://issues.apache.org/jira/browse/HBASE-12361
> Thanks so much!
> Gokul Balakrishnan.

Best regards,

   - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message