hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: How can I set the num of mappers when I use hbase RowCounter on Yarn?
Date Mon, 20 Oct 2014 13:58:19 GMT
RowCounter calls:

    TableMapReduceUtil.initTableMapperJob(tableName, scan,

      RowCounterMapper.class, ImmutableBytesWritable.class, Result.class,
job);

which uses TableInputFormat.

   * Calculates the splits that will serve as input for the map tasks. The

   * number of splits matches the number of regions in a table.

FYI

On Mon, Oct 20, 2014 at 6:23 AM, Shahab Yunus <shahab.yunus@gmail.com>
wrote:

> Have you tried setting the following property through the command line?
>
> -D mapreduce.job.mappers
>
>
> Regards,
>
> Shahab
>
>
> On Mon, Oct 20, 2014 at 2:24 AM, liubiyi@inspur.com <liubiyi@inspur.com>
> wrote:
>
> >  Hello,
> > I used hbase Rowcounter on yarn , but the num of mappers was  1, and the
> > progress was 0%.
> > The input data was quit large, about  587GB.
> > So how can I set the num of mappers when I use hbase RowCounter on yarn
> to
> > make it faster?
> > The commands I used were below:
> >
> > hadoop jar  hbase-server-0.99.0-hadoop2-hadoop.jar importtsv
> -Dimporttsv.columns=HBASE_ROW_KEY,cf:c1,cf:c2,cf:c3,cf:c4,cf:c5,cf:c6,cf:c7
> -Dimporttsv.separator="," -Dimporttsv.bulk.output=/inspur/output1 test
> /inspur/input/data/test.txt
> >
> > hadoop jar hbase-server-0.99.0-hadoop2-hadoop.jar completebulkload
> > /inspur/output1 test
> >
> >  hbase org.apache.hadoop.hbase.mapreduce.RowCounter test
> >
> > ------------------------------
> >         -----------------
> > 刘璧怡
> > 系统软件部研发三处
> >  浪潮(北京)电子信息产业有限公司
> > TEL:010-61643876/18500682190
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message