hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tim Sell" <trs...@gmail.com>
Subject Re: Issue with NotServingRegionException
Date Tue, 05 Aug 2008 17:10:36 GMT
I wish I did. My client just stopped with no error message, which is
my fault somewhere.
The thrift server was spitting useful stuff out, but I forgot to
pipe/log it to file and most of it disappeared out the top of my
terminal's buffer.

I think my thrift client timed out and just stopped, but the thrift
server kept trying to find the region, for hours.. I'm not sure that
makes sense, but I don't think anything else was trying to use the
thrift interface (and definitely not for the table "catalogue").
The call in the trace back mutateRows could only have been initiated
by my client.
The tail of it, (some two hours after I think the region got lost) was:

08/08/05 13:56:28 DEBUG client.HConnectionManager$TableServers:
Removed .META.,,1 from cache because of
catalogue,21370921,99999999999999
08/08/05 13:56:28 DEBUG client.HConnectionManager$TableServers: Found
ROOT REGION => {NAME => '-ROOT-,,0', STARTKEY => '', ENDKEY => '',
ENCODED => 70236052, TABLE => {{NAME => '-ROOT-', IS_ROOT => 'true',
IS_META => 'true', FAMILIES => [{NAME => 'info', BLOOMFILTER =>
'false', COMPRESSION => 'NONE', VERSIONS => '1', LENGTH =>
'2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE =>
'false'}]}}
08/08/05 13:56:38 DEBUG client.HConnectionManager$TableServers:
reloading table servers because: HRegionInfo was null or empty in
.META.
08/08/05 13:56:38 DEBUG client.HConnectionManager$TableServers:
Removed .META.,,1 from cache because of
catalogue,21370921,99999999999999
08/08/05 13:56:38 DEBUG client.HConnectionManager$TableServers: Found
ROOT REGION => {NAME => '-ROOT-,,0', STARTKEY => '', ENDKEY => '',
ENCODED => 70236052, TABLE => {{NAME => '-ROOT-', IS_ROOT => 'true',
IS_META => 'true', FAMILIES => [{NAME => 'info', BLOOMFILTER =>
'false', COMPRESSION => 'NONE', VERSIONS => '1', LENGTH =>
'2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE =>
'false'}]}}
08/08/05 13:56:58 DEBUG client.HConnectionManager$TableServers:
reloading table servers because: HRegionInfo was null or empty in
.META.
08/08/05 13:56:58 DEBUG client.HConnectionManager$TableServers:
Removed .META.,,1 from cache because of
catalogue,21370921,99999999999999
08/08/05 13:56:58 DEBUG client.HConnectionManager$TableServers: Found
ROOT REGION => {NAME => '-ROOT-,,0', STARTKEY => '', ENDKEY => '',
ENCODED => 70236052, TABLE => {{NAME => '-ROOT-', IS_ROOT => 'true',
IS_META => 'true', FAMILIES => [{NAME => 'info', BLOOMFILTER =>
'false', COMPRESSION => 'NONE', VERSIONS => '1', LENGTH =>
'2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE =>
'false'}]}}
08/08/05 13:57:38 DEBUG client.HConnectionManager$TableServers:
reloading table servers because: HRegionInfo was null or empty in
.META.
08/08/05 13:57:38 DEBUG client.HConnectionManager$TableServers:
Removed .META.,,1 from cache because of
catalogue,21370921,99999999999999
08/08/05 13:57:38 DEBUG client.HConnectionManager$TableServers: Found
ROOT REGION => {NAME => '-ROOT-,,0', STARTKEY => '', ENDKEY => '',
ENCODED => 70236052, TABLE => {{NAME => '-ROOT-', IS_ROOT => 'true',
IS_META => 'true', FAMILIES => [{NAME => 'info', BLOOMFILTER =>
'false', COMPRESSION => 'NONE', VERSIONS => '1', LENGTH =>
'2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE =>
'false'}]}}
08/08/05 13:58:58 DEBUG client.HConnectionManager$TableServers:
reloading table servers because: HRegionInfo was null or empty in
.META.
08/08/05 13:58:58 DEBUG client.HConnectionManager$TableServers:
Removed .META.,,1 from cache because of
catalogue,21370921,99999999999999
08/08/05 13:58:58 DEBUG client.HConnectionManager$TableServers: Found
ROOT REGION => {NAME => '-ROOT-,,0', STARTKEY => '', ENDKEY => '',
ENCODED => 70236052, TABLE => {{NAME => '-ROOT-', IS_ROOT => 'true',
IS_META => 'true', FAMILIES => [{NAME => 'info', BLOOMFILTER =>
'false', COMPRESSION => 'NONE', VERSIONS => '1', LENGTH =>
'2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE =>
'false'}]}}
08/08/05 14:01:38 DEBUG client.HConnectionManager$TableServers:
reloading table servers because: HRegionInfo was null or empty in
.META.
08/08/05 14:01:38 DEBUG client.HConnectionManager$TableServers:
Removed .META.,,1 from cache because of
catalogue,21370921,99999999999999
08/08/05 14:01:38 DEBUG client.HConnectionManager$TableServers: Found
ROOT REGION => {NAME => '-ROOT-,,0', STARTKEY => '', ENDKEY => '',
ENCODED => 70236052, TABLE => {{NAME => '-ROOT-', IS_ROOT => 'true',
IS_META => 'true', FAMILIES => [{NAME => 'info', BLOOMFILTER =>
'false', COMPRESSION => 'NONE', VERSIONS => '1', LENGTH =>
'2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE =>
'false'}]}}
java.lang.NullPointerException
        at org.apache.hadoop.hbase.client.ServerCallable.getServerName(ServerCallable.java:63)
        at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:892)
        at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:1168)
        at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:1187)
        at org.apache.hadoop.hbase.thrift.ThriftServer$HBaseHandler.mutateRowsTs(ThriftServer.java:474)
        at org.apache.hadoop.hbase.thrift.ThriftServer$HBaseHandler.mutateRows(ThriftServer.java:437)
        at org.apache.hadoop.hbase.thrift.generated.Hbase$Processor$mutateRows.process(Hbase.java:1668)
        at org.apache.hadoop.hbase.thrift.generated.Hbase$Processor.process(Hbase.java:1348)
        at com.facebook.thrift.server.TThreadPoolServer$WorkerProcess.run(Unknown
Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
        at java.lang.Thread.run(Thread.java:619)



2008/8/5 Andrew Purtell <apurtell@yahoo.com>:
> Hi Tim,
>
> By chance do you have a log from the Thrift client?
>
>   - Andy
>
>
> --- On Tue, 8/5/08, Tim Sell <trsell@gmail.com> wrote:
>
>> From: Tim Sell <trsell@gmail.com>
>> Subject: Re: Issue with NotServingRegionException
>> To: hbase-user@hadoop.apache.org
>> Date: Tuesday, August 5, 2008, 8:40 AM
> [...]
>> We have the same setup as Nickolai, 3 machines each with a
>> region server, with one machine acting as the master as well.
>>
>> We were bootstrapping data into it. After a while in one of
>> the region server logs we got an info:
>> NotServingRegionException when a batchUpdate was called.
>> This occurred directly after the region was closed because
>> of a split.
>>
>> Then eventually my thrift client crashed because thrift
>> wasn't returning.
> [...]
>
>
>
>
>

Mime
View raw message