hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Gray <jg...@fb.com>
Subject RE: Hive HBase integration scan failing
Date Fri, 10 Dec 2010 09:52:01 GMT
Hey,

Need some more info.

Can you paste logs from the MR tasks that fail?  What's going on in the cluster while the
MR job is running (cpu, io-wait, memory, etc)?

And what is the setup of your cluster... how many nodes, specs of nodes (cores, memory, RS
heap), and then how many concurrent map tasks you have per node.

JG

> -----Original Message-----
> From: vlisovsky [mailto:vlisovsky@gmail.com]
> Sent: Thursday, December 09, 2010 10:49 PM
> To: user@hbase.apache.org
> Subject: Hive HBase integration scan failing
> 
> Hi Guys,
> 
> > Wonder if  anybody could shed some light on how to reduce the load on
> > HBase cluster when running a full scan.
> > The need is to dump everything I have in HBase and into a Hive table.
> > The HBase data size is around 500g.
> > The job creates 9000 mappers, after about 1000 maps things go south
> > every time..
> > If I run below insert it runs for about 30 minutes then starts
> > bringing down HBase cluster after which region servers need to be
> restarted..
> > Wonder if there is a way to throttle it somehow or otherwise if there
> > is any other method of getting structured data out?
> > Any help is appreciated,
> > Thanks,
> > -Vitaly
> >
> > create external table hbase_linked_table (
> > mykey        string,
> > info        map<string, string>,
> > )
> > STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
> > WITH
> > SERDEPROPERTIES ("hbase.columns.mapping" = ":key,info:")
> TBLPROPERTIES
> > ("hbase.table.name" = "hbase_table2");
> >
> > set hive.exec.compress.output=true;
> > set io.seqfile.compression.type=BLOCK;
> > set mapred.output.compression.type=BLOCK;
> > set
> >
> mapred.output.compression.codec=org.apache.hadoop.io.compress.GzipCo
> de
> > c;
> >
> > set mapred.reduce.tasks=40;
> > set mapred.map.tasks=25;
> >
> > INSERT overwrite table tmp_hive_destination select * from
> > hbase_linked_table;
> >

Mime
View raw message