hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kireet <kir...@feedly.com>
Subject Re: HConnectionManager$HConnectionImplementation.locateRegionInMeta
Date Thu, 30 May 2013 18:28:57 GMT


Thanks, will give it a shot. So I should download 0.94.7 (latest stable) 
and run the patch tool on top with the backport? This is a little new to me.

Also, I was looking at the stack below. From my reading of the code, the 
HTable.batch() call will always cause the prefetch call to occur, which 
will cause a new HTable object to get created. The constructor used in 
creating a new thread pool. So every call to batch will create a new 
thread? Or the HTable's thread pool never gets used as the pool is only 
used for writes? I think I am missing something but just want to confirm.

Thanks
Kireet

On 5/30/13 12:48 PM, Himanshu Vashishtha wrote:
> bq. Anoop attached backported patch in HBASE-8655. It should go into
> 0.94.9, the next release - current is 0.94.8
>
> In case you want it sooner, you can apply 8655 patch and test/verify it.
>
> Thanks,
> Himanshu
>
>
> On Thu, May 30, 2013 at 7:26 AM, Ted Yu <yuzhihong-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
wrote:
>
>> Anoop attached backported patch in HBASE-8655
>>
>> It should go into 0.94.9, the next release - current is 0.94.8
>>
>> Cheers
>>
>> On Thu, May 30, 2013 at 7:01 AM, Kireet <kireet-Teh5dPVPL8nQT0dZR+AlfA@public.gmane.org>
wrote:
>>
>>>
>>>
>>> How long do backports typically take? We have to go live in a month ready
>>> or not. Thanks for the quick replies Anoop and Ted.
>>>
>>> --Kireet
>>>
>>>
>>> On 5/30/13 9:20 AM, Ted Yu wrote:
>>>
>>>> 0.95 client is not compatible with 0.94 cluster. So you cannot use 0.95
>>>> client.
>>>>
>>>> Cheers
>>>>
>>>> On May 30, 2013, at 6:12 AM, Kireet <kireet-Teh5dPVPL8nQT0dZR+**
>>>> AlfA-XMD5yJDbdMReXY1tMh2IBg@public.gmane.org <kireet-Teh5dPVPL8nQT0dZR%2BAlfA-XMD5yJDbdMReXY1tMh2IBg@public.gmane.org
>>>>
>>>> wrote:
>>>>
>>>>
>>>>>
>>>>> Would there be a problem if our cluster is 0.94 and we use a 0.95
>> client?
>>>>>
>>>>> I am not familiar with the HBase code base, but I did a dump of the
>>>>> thread that is actually running (below). It seems like it is related
>> to the
>>>>> issue you mentioned as the running thread is doing the prefetch logic.
>>>>> Would pre-splitting tables help here? We are doing some performance
>> tests
>>>>> and essentially starting from an empty instance.
>>>>>
>>>>> java.lang.Thread.State: WAITING (on object monitor)
>>>>> at java.lang.Object.wait(Native Method)
>>>>> at java.lang.Object.wait(Object.**java:503)
>>>>> at org.apache.zookeeper.**ClientCnxn.submitRequest(**
>>>>> ClientCnxn.java:1309)
>>>>> - locked <0x00000000e10cf830> (a org.apache.zookeeper.**
>>>>> ClientCnxn$Packet)
>>>>> at org.apache.zookeeper.**ZooKeeper.exists(ZooKeeper.**java:1036)
>>>>> at org.apache.hadoop.hbase.**zookeeper.**RecoverableZooKeeper.exists(**
>>>>> RecoverableZooKeeper.java:172)
>>>>> at org.apache.hadoop.hbase.**zookeeper.ZKUtil.checkExists(**
>>>>> ZKUtil.java:450)
>>>>> at org.apache.hadoop.hbase.**zookeeper.**ZooKeeperNodeTracker.**
>>>>> checkIfBaseNodeAvailable(**ZooKeeperNodeTracker.java:208)
>>>>> at org.apache.hadoop.hbase.**zookeeper.RootRegionTracker.**
>>>>> waitRootRegionLocation(**RootRegionTracker.java:77)
>>>>> at org.apache.hadoop.hbase.**client.HConnectionManager$**
>>>>> HConnectionImplementation.**locateRegion(**HConnectionManager.java:874)
>>>>> at org.apache.hadoop.hbase.**client.HConnectionManager$**
>>>>> HConnectionImplementation.**locateRegionInMeta(**
>>>>> HConnectionManager.java:987)
>>>>> at org.apache.hadoop.hbase.**client.HConnectionManager$**
>>>>> HConnectionImplementation.**locateRegion(**HConnectionManager.java:885)
>>>>> at org.apache.hadoop.hbase.**client.HConnectionManager$**
>>>>> HConnectionImplementation.**locateRegion(**HConnectionManager.java:846)
>>>>> at org.apache.hadoop.hbase.**client.HTable.finishSetup(**
>>>>> HTable.java:234)
>>>>> at org.apache.hadoop.hbase.**client.HTable.<init>(HTable.**java:174)
>>>>> at org.apache.hadoop.hbase.**client.MetaScanner.metaScan(**
>>>>> MetaScanner.java:160)
>>>>> at org.apache.hadoop.hbase.**client.MetaScanner.access$000(**
>>>>> MetaScanner.java:54)
>>>>> at org.apache.hadoop.hbase.**client.MetaScanner$1.connect(**
>>>>> MetaScanner.java:133)
>>>>> at org.apache.hadoop.hbase.**client.MetaScanner$1.connect(**
>>>>> MetaScanner.java:130)
>>>>> at org.apache.hadoop.hbase.**client.HConnectionManager.**
>>>>> execute(HConnectionManager.**java:383)
>>>>> at org.apache.hadoop.hbase.**client.MetaScanner.metaScan(**
>>>>> MetaScanner.java:130)
>>>>> at org.apache.hadoop.hbase.**client.MetaScanner.metaScan(**
>>>>> MetaScanner.java:105)
>>>>> at org.apache.hadoop.hbase.**client.HConnectionManager$**
>>>>> HConnectionImplementation.**prefetchRegionCache(**
>>>>> HConnectionManager.java:947)
>>>>> at org.apache.hadoop.hbase.**client.HConnectionManager$**
>>>>> HConnectionImplementation.**locateRegionInMeta(**
>>>>> HConnectionManager.java:1002)
>>>>> - locked <0x00000000e882f778> (a java.lang.Object)
>>>>> at org.apache.hadoop.hbase.**client.HConnectionManager$**
>>>>> HConnectionImplementation.**locateRegion(**HConnectionManager.java:889)
>>>>> at org.apache.hadoop.hbase.**client.HConnectionManager$**
>>>>> HConnectionImplementation.**locateRegion(**HConnectionManager.java:846)
>>>>> at org.apache.hadoop.hbase.**client.HConnectionManager$**
>>>>> HConnectionImplementation.**processBatchCallback(**
>>>>> HConnectionManager.java:1533)
>>>>> at org.apache.hadoop.hbase.**client.HConnectionManager$**
>>>>>
>> HConnectionImplementation.**processBatch(**HConnectionManager.java:1418)
>>>>> at org.apache.hadoop.hbase.**client.HTable.batch(HTable.**java:702)
>>>>> at org.apache.hadoop.hbase.**client.HTable.get(HTable.java:**671)
>>>>> at org.apache.hadoop.hbase.**client.HTablePool$**
>>>>> PooledHTable.get(HTablePool.**java:367)
>>>>>
>>>>> Thanks
>>>>> Kireet
>>>>>
>>>>> On 5/30/13 12:16 AM, Anoop John wrote:
>>>>>
>>>>>> Can you have a look at issue HBASE-8476?  Seems related?  A fix is
>>>>>> available in HBASE-8346's patch..
>>>>>>
>>>>>> -Anoop-
>>>>>>
>>>>>> On Thu, May 30, 2013 at 9:21 AM, Kireet <kireet-Teh5dPVPL8nQT0dZR+**
>>>>>> AlfA-XMD5yJDbdMReXY1tMh2IBg@public.gmane.org<
>> kireet-Teh5dPVPL8nQT0dZR%2BAlfA-XMD5yJDbdMReXY1tMh2IBg@public.gmane.org>>
>>>>>> wrote:
>>>>>>
>>>>>>   We are running hbase 0.94.6 in a concurrent environment and we
are
>>>>>>> seeing
>>>>>>> the majority of our code stuck in this method at the synchronized
>> block
>>>>>>> (line 997). Is there some configuration we can change to avoid
it? It
>>>>>>> seems
>>>>>>> like we could disable prefetching, but that sounds like it could
>> cause
>>>>>>> other problems?
>>>>>>>
>>>>>>>    - org.apache.hadoop.hbase.****client.HConnectionManager$**
>>>>>>> HConnectionImplementation.****locateRegionInMeta(byte[], byte[],
>>>>>>> byte[],
>>>>>>> boolean, java.lang.Object, boolean) @bci=149, line=997 (Compiled
>> frame)
>>>>>>>    - org.apache.hadoop.hbase.****client.HConnectionManager$**
>>>>>>> HConnectionImplementation.****locateRegion(byte[], byte[], boolean,
>>>>>>> boolean) @bci=212, line=889 (Compiled frame)
>>>>>>>    - org.apache.hadoop.hbase.****client.HConnectionManager$**
>>>>>>> HConnectionImplementation.****locateRegion(byte[], byte[]) @bci=5,
>>>>>>> line=846
>>>>>>> (Compiled frame)
>>>>>>>    - org.apache.hadoop.hbase.****client.HConnectionManager$**
>>>>>>>
>> HConnectionImplementation.****processBatchCallback(java.****util.List,
>>>>>>> byte[], java.util.concurrent.****ExecutorService, java.lang.Object[],
>>>>>>> org.apache.hadoop.hbase.****client.coprocessor.Batch$****Callback)
>>>>>>> @bci=194,
>>>>>>> line=1533 (Compiled frame)
>>>>>>>    - org.apache.hadoop.hbase.****client.HConnectionManager$**
>>>>>>> HConnectionImplementation.****processBatch(java.util.List, byte[],
>>>>>>> java.util.concurrent.****ExecutorService, java.lang.Object[])
>> @bci=30,
>>>>>>> line=1418 (Compiled frame)
>>>>>>>    - org.apache.hadoop.hbase.****client.HTable.batch(java.util.**
>>>>>>> **List)
>>>>>>> @bci=24, line=702 (Compiled frame)
>>>>>>>    - org.apache.hadoop.hbase.****client.HTable.get(java.util.****List)
>>>>>>> @bci=2,
>>>>>>> line=671 (Compiled frame)
>>>>>>>    - org.apache.hadoop.hbase.****client.HTablePool$**
>>>>>>> PooledHTable.get(java.util.****List) @bci=5, line=367 (Compiled
>> frame)
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>


Mime
View raw message