incubator-hcatalog-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alan Gates <>
Subject Re: Hadoop streaming with HCatalog
Date Thu, 22 Aug 2013 15:41:38 GMT
Sorry for the late reply.  I am not aware of anyone working on making Hadoop streaming work
with HCatalog.  However, we have always believed it should, and if anyone wanted to take up
the task we'd be happy to get the work reviewed and checked in.


On Jul 24, 2013, at 5:34 AM, Ajay Singh wrote:

> Hi all,
> I have been playing around with HCatalog for last few days. I love it. Great work!!!
> Is the HCatalog team planning to enhance hadoop-streaming to use HCatalog in near future?
Can we expect it by this year?
> I looked at the hadoop-streaming code and was wondering how one would enhance hadoop-streaming
to consume from / write to  HCatalog managed tables. 
> Hadoop-streaming launches a map-reduce job. The MapTask of this job manages the communication
with the external non-java mapper (through stdin and stdout). The ReduceTask does the same
with the non-java reducer. I doubt anything needs to be changed here. What needs changed is
the input to MapTask and output from ReduceTask.
> So if we modify the MapTask/ReduceTask to read from / write to a HCatalog table, that
should do it right? Since HCatalog already supports M/R, this should be just about re-writing
the streaming job using HCatInputFormat, HCatOutputFormat, HCatRecord etc. I noticed that
hadoop-streaming uses old map-red API (org.apache.mapred). Do we need to move to new map-reduce
API (org.apache.mapreduce) to use HCatalog?
> Thanks
> Ajay

NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

View raw message