hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stack <st...@duboce.net>
Subject Re: HBase Read and Write Issues in Mutlithreaded Environments
Date Wed, 29 Jun 2011 05:45:56 GMT
Can you upgrade?  That release is > 18 months old.  A bunch has
happened in the meantime.

For retries exhausted, check whats going on on the remote regionserver
that you are trying to write too.  Its probably struggling and thats
why requests are not going through -- or the client missed the fact
that region moved (all stuff that should be working better in latest
hbase).

St.Ack

On Tue, Jun 28, 2011 at 9:51 PM, Srikanth P. Shreenivas
<Srikanth_Shreenivas@mindtree.com> wrote:
> Hi,
>
> We are using HBase 0.20.3 (hbase-0.20-0.20.3-1.cloudera.noarch.rpm) cluster in distributed
mode with Hadoop 0.20.2 (hadoop-0.20-0.20.2+320-1.noarch).
> We are using pretty much default configuration, and only thing we have customized is
that we have allocated 4GB RAM in /etc/hbase-0.20/conf/hbase-env.sh
>
> In our setup, we have a web application that reads a record from HBase and writes a record
as part of each web request.   The application is hosted in Apache Tomcat 7 and is a stateless
web application providing a REST-like web service API.
>
> We are observing that our reads and writes times out once in a  while.  This happens
more for writes.
> We see below exception in our application logs:
>
>
> Exception Type 1 - During Get:
> ---------------------------------------
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server
10.1.68.36:60020 for region employeedata,be8784ac8b57c45625a03d52be981b88097c2fdc,1308657957879,
row 'd51b74eb05e07f96cee0ec556f5d8d161e3281f3', but failed after 10 attempts.
> Exceptions:
> java.io.IOException: Call to /10.1.68.36:60020 failed on local exception: java.nio.channels.ClosedByInterruptException
> java.nio.channels.ClosedByInterruptException
> java.nio.channels.ClosedByInterruptException
> java.nio.channels.ClosedByInterruptException
> java.nio.channels.ClosedByInterruptException
> java.nio.channels.ClosedByInterruptException
> java.nio.channels.ClosedByInterruptException
> java.nio.channels.ClosedByInterruptException
> java.nio.channels.ClosedByInterruptException
> java.nio.channels.ClosedByInterruptException
>
>        at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:1048)
>        at org.apache.hadoop.hbase.client.HTable.get(HTable.java:417)
>     <snip>
>
> Exception  Type 2 - During Put:
> ---------------------------------------------
> Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact
region server 10.1.68.34:60020 for region audittable,,1309183872019, row '2a012017120f80a801b28f5f66a83dc2a8882d1b',
but failed after 10 attempts.
> Exceptions:
> java.io.IOException: Call to /10.1.68.34:60020 failed on local exception: java.nio.channels.ClosedByInterruptException
> java.io.IOException: Call to /10.1.68.34:60020 failed on local exception: java.nio.channels.ClosedByInterruptException
> java.io.IOException: Call to /10.1.68.34:60020 failed on local exception: java.nio.channels.ClosedByInterruptException
> java.io.IOException: Call to /10.1.68.34:60020 failed on local exception: java.nio.channels.ClosedByInterruptException
> java.io.IOException: Call to /10.1.68.34:60020 failed on local exception: java.nio.channels.ClosedByInterruptException
> java.io.IOException: Call to /10.1.68.34:60020 failed on local exception: java.nio.channels.ClosedByInterruptException
> java.io.IOException: Call to /10.1.68.34:60020 failed on local exception: java.nio.channels.ClosedByInterruptException
> java.io.IOException: Call to /10.1.68.34:60020 failed on local exception: java.nio.channels.ClosedByInterruptException
> java.io.IOException: Call to /10.1.68.34:60020 failed on local exception: java.nio.channels.ClosedByInterruptException
> java.io.IOException: Call to /10.1.68.34:60020 failed on local exception: java.nio.channels.ClosedByInterruptException
>
>        at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:1048)
>        at org.apache.hadoop.hbase.client.HConnectionManager$TableServers$3.doCall(HConnectionManager.java:1239)
>        at org.apache.hadoop.hbase.client.HConnectionManager$TableServers$Batch.process(HConnectionManager.java:1161)
>        at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.processBatchOfRows(HConnectionManager.java:1247)
>        at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:609)
>        at org.apache.hadoop.hbase.client.HTable.put(HTable.java:474)
>     <snip>
>
> Any inputs on why this is happening, or how to rectify it will be of immense help.
>
> Thanks,
> Srikanth
>
>
>
> Srikanth P Shreenivas|Principal Consultant | MindTree Ltd.|Global Village, RVCE Post,
Mysore Road, Bangalore-560 059, INDIA|Voice +91 80 26264000 / Fax +91 80 2626 4100| Mob: 9880141059|email:
srikanth_shreenivas@mindtree.com<mailto:asha_venkatesh@mindtree.com> |www.mindtree.com<http://www.mindtree.com/>
|
>
>
> ________________________________
>
> http://www.mindtree.com/email/disclaimer.html
>

Mime
View raw message