hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stack <st...@duboce.net>
Subject Re: HBase Read and Write Issues in Mutlithreaded Environments
Date Wed, 29 Jun 2011 14:17:48 GMT
Go to CDH3 if you can.  CDH2 is also old.
St.Ack

On Wed, Jun 29, 2011 at 7:15 AM, Srikanth P. Shreenivas
<Srikanth_Shreenivas@mindtree.com> wrote:
> Thanks St. Ack for the inputs.
>
> Will upgrading to CDH3 help or is there a version within CDH2 that you recommend we should
upgrade to?
>
> Regards,
> Srikanth
>
> -----Original Message-----
> From: saint.ack@gmail.com [mailto:saint.ack@gmail.com] On Behalf Of Stack
> Sent: Wednesday, June 29, 2011 11:16 AM
> To: user@hbase.apache.org
> Subject: Re: HBase Read and Write Issues in Mutlithreaded Environments
>
> Can you upgrade?  That release is > 18 months old.  A bunch has
> happened in the meantime.
>
> For retries exhausted, check whats going on on the remote regionserver
> that you are trying to write too.  Its probably struggling and thats
> why requests are not going through -- or the client missed the fact
> that region moved (all stuff that should be working better in latest
> hbase).
>
> St.Ack
>
> On Tue, Jun 28, 2011 at 9:51 PM, Srikanth P. Shreenivas
> <Srikanth_Shreenivas@mindtree.com> wrote:
>> Hi,
>>
>> We are using HBase 0.20.3 (hbase-0.20-0.20.3-1.cloudera.noarch.rpm) cluster in distributed
mode with Hadoop 0.20.2 (hadoop-0.20-0.20.2+320-1.noarch).
>> We are using pretty much default configuration, and only thing we have customized
is that we have allocated 4GB RAM in /etc/hbase-0.20/conf/hbase-env.sh
>>
>> In our setup, we have a web application that reads a record from HBase and writes
a record as part of each web request.   The application is hosted in Apache Tomcat 7 and
is a stateless web application providing a REST-like web service API.
>>
>> We are observing that our reads and writes times out once in a  while.  This happens
more for writes.
>> We see below exception in our application logs:
>>
>>
>> Exception Type 1 - During Get:
>> ---------------------------------------
>> org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region
server 10.1.68.36:60020 for region employeedata,be8784ac8b57c45625a03d52be981b88097c2fdc,1308657957879,
row 'd51b74eb05e07f96cee0ec556f5d8d161e3281f3', but failed after 10 attempts.
>> Exceptions:
>> java.io.IOException: Call to /10.1.68.36:60020 failed on local exception: java.nio.channels.ClosedByInterruptException
>> java.nio.channels.ClosedByInterruptException
>> java.nio.channels.ClosedByInterruptException
>> java.nio.channels.ClosedByInterruptException
>> java.nio.channels.ClosedByInterruptException
>> java.nio.channels.ClosedByInterruptException
>> java.nio.channels.ClosedByInterruptException
>> java.nio.channels.ClosedByInterruptException
>> java.nio.channels.ClosedByInterruptException
>> java.nio.channels.ClosedByInterruptException
>>
>>        at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:1048)
>>        at org.apache.hadoop.hbase.client.HTable.get(HTable.java:417)
>>     <snip>
>>
>> Exception  Type 2 - During Put:
>> ---------------------------------------------
>> Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact
region server 10.1.68.34:60020 for region audittable,,1309183872019, row '2a012017120f80a801b28f5f66a83dc2a8882d1b',
but failed after 10 attempts.
>> Exceptions:
>> java.io.IOException: Call to /10.1.68.34:60020 failed on local exception: java.nio.channels.ClosedByInterruptException
>> java.io.IOException: Call to /10.1.68.34:60020 failed on local exception: java.nio.channels.ClosedByInterruptException
>> java.io.IOException: Call to /10.1.68.34:60020 failed on local exception: java.nio.channels.ClosedByInterruptException
>> java.io.IOException: Call to /10.1.68.34:60020 failed on local exception: java.nio.channels.ClosedByInterruptException
>> java.io.IOException: Call to /10.1.68.34:60020 failed on local exception: java.nio.channels.ClosedByInterruptException
>> java.io.IOException: Call to /10.1.68.34:60020 failed on local exception: java.nio.channels.ClosedByInterruptException
>> java.io.IOException: Call to /10.1.68.34:60020 failed on local exception: java.nio.channels.ClosedByInterruptException
>> java.io.IOException: Call to /10.1.68.34:60020 failed on local exception: java.nio.channels.ClosedByInterruptException
>> java.io.IOException: Call to /10.1.68.34:60020 failed on local exception: java.nio.channels.ClosedByInterruptException
>> java.io.IOException: Call to /10.1.68.34:60020 failed on local exception: java.nio.channels.ClosedByInterruptException
>>
>>        at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:1048)
>>        at org.apache.hadoop.hbase.client.HConnectionManager$TableServers$3.doCall(HConnectionManager.java:1239)
>>        at org.apache.hadoop.hbase.client.HConnectionManager$TableServers$Batch.process(HConnectionManager.java:1161)
>>        at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.processBatchOfRows(HConnectionManager.java:1247)
>>        at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:609)
>>        at org.apache.hadoop.hbase.client.HTable.put(HTable.java:474)
>>     <snip>
>>
>> Any inputs on why this is happening, or how to rectify it will be of immense help.
>>
>> Thanks,
>> Srikanth
>>
>>
>>
>> Srikanth P Shreenivas|Principal Consultant | MindTree Ltd.|Global Village, RVCE Post,
Mysore Road, Bangalore-560 059, INDIA|Voice +91 80 26264000 / Fax +91 80 2626 4100| Mob: 9880141059|email:
srikanth_shreenivas@mindtree.com<mailto:asha_venkatesh@mindtree.com> |www.mindtree.com<http://www.mindtree.com/>
|
>>
>>
>> ________________________________
>>
>> http://www.mindtree.com/email/disclaimer.html
>>
>

Mime
View raw message