hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Anoop Sam John <anoo...@huawei.com>
Subject RE: HBase Checksum
Date Fri, 01 Feb 2013 05:51:07 GMT
Hi Robert
          When HDFS is doing the local short circuit read, it will use BlockReaderLocal class
for reading.  There should be some logs at the DFS client side (RS) which tells abt creating
new BlockReaderLocal .  If you can see this then sure the local read is happening.

Also check DN log.  If local read happening, then you will not see  read request related logs
for the HFile at the DN side.  
You check your no# of HFiles and names for checking the logs

Are you sure that when you tested, u have data locality? Region movements across RSs can break
the full data locality.

From: Robert Dyer [psybers@gmail.com]
Sent: Friday, February 01, 2013 11:10 AM
To: Hbase-User
Subject: Re: HBase Checksum

Not trying to hijack your thread here...

But can you verify via logs that the shortcircuit is working?  Because I
enabled shortcircuit but I sure didn't see any performance increase.

I haven't tried enabling hbase checksum yet but I'd like to be able to
verify that works too.

On Thu, Jan 31, 2013 at 9:55 PM, Anoop Sam John <anoopsj@huawei.com> wrote:

> You can check with HDFS level logs whether the checksum meta file is
> getting read to the DFS client? In the HBase handled checksum, this should
> not happen.
> Have you noticed any perf gain when you configure the HBase handled
> checksum option?
> -Anoop-
> ________________________________________
> From: Jean-Marc Spaggiari [jean-marc@spaggiari.org]
> Sent: Friday, February 01, 2013 4:16 AM
> To: user
> Subject: HBase Checksum
> Hi,
> I have activated shortcircuit and checksum and I would like to get a
> confirmation that it's working fine.
> So I have activated short circuit first and saw a 40% improvement of
> the MR rowcount job. So I guess it's working fine.
> Now, I'm configuring the checksum option, and I'm wondering how I can
> do to validate that it's taken into consideration and used, or not. Is
> there a way to see that?
> Thanks,
> JM
View raw message