hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: assert in hbase 1.4
Date Wed, 03 Jan 2018 22:55:44 GMT
Thanks, Clément

On Wed, Jan 3, 2018 at 2:54 PM, Clément Guillaume <clement@guillaume.bzh>
wrote:

> Done https://issues.apache.org/jira/browse/HBASE-19700
>
> 2017-12-31 8:16 GMT-08:00 Ted Yu <yuzhihong@gmail.com>:
>
>> Clément:
>> Since you have reproduced the assertion error, can you log a JIRA ?
>>
>> Thanks
>>
>> On Sun, Dec 31, 2017 at 7:46 AM, Ted Yu <yuzhihong@gmail.com> wrote:
>>
>>> I modified the test so that the unit test runs against branch-1.4 :
>>>
>>> +      Configuration hbaseConfig = TEST_UTIL.getConfiguration();
>>> +      //hbaseConfig.set(HConstants.ZOOKEEPER_QUORUM,
>>> "hbase123.docker");
>>>
>>> However the test runs to completion without assertion - meaning the
>>> assertion wouldn't be triggered in homogeneous cluster.
>>>
>>> Here were the last few lines in the test output:
>>>
>>> 2017-12-31 07:33:36,865 INFO  [Time-limited test]
>>> client.HBaseAdmin$CreateTableFuture(793): Created AssertionTest
>>> table created
>>> 2017-12-31 07:33:36,871 DEBUG [Time-limited test]
>>> client.ClientScanner(248): Advancing internal scanner to startKey at
>>> '\x01\x02\x03\x04\x05\x06\x07', inclusive
>>> 2017-12-31 07:33:36,873 DEBUG [hconnection-0x6e0982ab-shared--pool24-t3]
>>> ipc.RpcConnection(126): Use SIMPLE authentication for service
>>> ClientService, sasl=false
>>> 2017-12-31 07:33:36,873 DEBUG [hconnection-0x6e0982ab-shared--pool24-t3]
>>> ipc.BlockingRpcConnection(431): Connecting to
>>> tyus-mbp.attlocal.net/192.168.1.64:61585
>>> 2017-12-31 07:33:36,875 DEBUG [RpcServer.listener,port=61585]
>>> ipc.RpcServer$Listener(920): RpcServer.listener,port=61585: connection from
>>> 192.168.1.64:61633; # active          connections: 3
>>> 2017-12-31 07:33:36,875 INFO  [RpcServer.reader=0,bindAddress=
>>> tyus-mbp.attlocal.net,port=61585] ipc.RpcServer$Connection(1809): Auth
>>> successful for tyu (auth:SIMPLE)
>>> 2017-12-31 07:33:36,875 INFO  [RpcServer.reader=0,bindAddress=
>>> tyus-mbp.attlocal.net,port=61585] ipc.RpcServer$Connection(1839):
>>> Connection from 192.168.1.64 port: 61633 with   version info: version:
>>> "1.4.1-SNAPSHOT" url: "git://TYus-MBP.attlocal.net/Users/tyu/1.4-hbase"
>>> revision: "a7383851b9e62f23eb6f06eb23f09967f636c985" user: "tyu" date:
>>> "Sun Dec  31 07:32:58 PST 2017" src_checksum: "39bdbbe40374e2094f53ce77af539671"
>>> version_major: 1 version_minor: 4
>>> 2017-12-31 07:33:36,917 INFO  [main] hbase.ResourceChecker(171): after:
>>> client.TestHCM#test Thread=325 (was 305)
>>>
>>> I assume you pointed the test at a 1.2.3 cluster which triggered the
>>> assertion.
>>>
>>> It seems the patch which introduced the assertion didn't take into
>>> account client running against older release cluster deployment.
>>>
>>> FYI
>>>
>>> On Sun, Dec 31, 2017 at 6:09 AM, Clément Guillaume <
>>> clement@guillaume.bzh> wrote:
>>>
>>>> I have consistent failure with this code (a table with pre-splits):
>>>>
>>>> @Test
>>>>> public void test() throws IOException{
>>>>> Configuration hbaseConfig = HBaseConfiguration.create();
>>>>> hbaseConfig.set(HConstants.ZOOKEEPER_QUORUM, "hbase123.docker");
>>>>> Connection connection = ConnectionFactory.createConnec
>>>>> tion(hbaseConfig);
>>>>>
>>>>
>>>>
>>>> TableName tableName = TableName.valueOf("AssertionTest");
>>>>> Admin admin = connection.getAdmin();
>>>>>
>>>>> if(!admin.tableExists(tableName)){
>>>>> HTableDescriptor htable = new HTableDescriptor(tableName);
>>>>> htable.addFamily(new HColumnDescriptor(new byte[]{(byte)'a'}));
>>>>> byte[][] splitPoints = {{1, 2, 3, 4, 5, 6, 7}};
>>>>> admin.createTable(htable, splitPoints);
>>>>> System.out.println("table created");
>>>>> }
>>>>>
>>>>> Table table = connection.getTable(tableName);
>>>>> ResultScanner scanner = table.getScanner(new Scan());
>>>>> scanner.iterator().hasNext(); // Exception thrown here
>>>>> }
>>>>
>>>>
>>>> 2017-12-30 5:04 GMT-08:00 Ted Yu <yuzhihong@gmail.com>:
>>>>
>>>>> bq. a table has been created with pre-splits
>>>>>
>>>>> Unless you record the initial number of regions somewhere, there is no
>>>>> built-in mechanism to provide the answer.
>>>>>
>>>>> Even if you cannot reproduce the assertion consistently, as long as
>>>>> the chance of reproduction is high, the unit test would still be useful.
>>>>>
>>>>> Cheers
>>>>>
>>>>> On Sat, Dec 30, 2017 at 3:31 AM, Clément Guillaume <
>>>>> clement@guillaume.bzh> wrote:
>>>>>
>>>>>> I dropped all the tables in my test cluster. Now I'm inconstantly
>>>>>> able to reproduce the issue. I'm digging more.
>>>>>>
>>>>>> I have a related question: is it possible to check if a table has
>>>>>> been created with pre-splits?
>>>>>>
>>>>>> I forgot to mention that I'm connecting to a version 1.2.3 cluster,
>>>>>> but I think this should be ok according to this post
>>>>>> http://mail-archives.apache.org/mod_mbox/hbase-dev/2017
>>>>>> 11.mbox/%3C5BAAC90F-31D8-4A5F-B9E4-BA61FF4CD40E%40gmail.com%3E
>>>>>>
>>>>>> 2017-12-25 17:10 GMT-08:00 Ted Yu <yuzhihong@gmail.com>:
>>>>>>
>>>>>>> I got the following if I use your code directly:
>>>>>>>
>>>>>>> testAssert(org.apache.hadoop.hbase.client.TestHCM)  Time elapsed:
>>>>>>> 0.096 sec  <<< ERROR!
>>>>>>> java.io.UncheckedIOException: org.apache.hadoop.hbase.TableNotFoundException:
>>>>>>> AssertionTest
>>>>>>> at org.apache.hadoop.hbase.client.TestHCM.testAssert(TestHCM.ja
>>>>>>> va:243)
>>>>>>>
>>>>>>> If I create the table at the beginning of the test, the test
passes
>>>>>>> without triggering assertion.
>>>>>>>
>>>>>>> I tried both master branch and branch-1.4
>>>>>>>
>>>>>>> As I mentioned previously, the assertion in your case was probably
>>>>>>> related to the formation of row keys in hbase:meta table.
>>>>>>> See if you can reproduce the assertion by presplitting the table,
>>>>>>> etc.
>>>>>>>
>>>>>>> Here is the diff I used based on branch-1.4
>>>>>>>
>>>>>>> diff --git a/hbase-server/src/test/java/o
>>>>>>> rg/apache/hadoop/hbase/client/TestHCM.java
>>>>>>> b/hbase-server/src/test/java/org/apache/hadoop/hbase/client/
>>>>>>> TestHCM.java
>>>>>>> index a5e3a65..2944c8e 100644
>>>>>>> --- a/hbase-server/src/test/java/org/apache/hadoop/hbase/client/
>>>>>>> TestHCM.java
>>>>>>> +++ b/hbase-server/src/test/java/org/apache/hadoop/hbase/client/
>>>>>>> TestHCM.java
>>>>>>> @@ -246,6 +246,18 @@ public class TestHCM {
>>>>>>>      TEST_UTIL.shutdownMiniCluster();
>>>>>>>    }
>>>>>>>
>>>>>>> +  @Test
>>>>>>> +  public void testAssert() throws IOException{
>>>>>>> +    Configuration hbaseConfig = TEST_UTIL.getConfiguration();
>>>>>>> +    final TableName tableName = TableName.valueOf("assert");
>>>>>>> +    TEST_UTIL.createTable(tableName, FAM_NAM).close();
>>>>>>> +    String zkQuorum = "hbase123.docker";
>>>>>>> +    //hbaseConfig.set(HConstants.ZOOKEEPER_QUORUM, zkQuorum);
>>>>>>> +    Connection connection = ConnectionFactory.createConnec
>>>>>>> tion(hbaseConfig);
>>>>>>> +    Table table = connection.getTable(tableName);
>>>>>>> +    ResultScanner scanner = table.getScanner(new Scan());
>>>>>>> +    scanner.iterator().hasNext(); // Exception thrown here
>>>>>>> +  }
>>>>>>>
>>>>>>>    private static int getHConnectionManagerCacheSize(){
>>>>>>>      return HConnectionTestingUtility.getConnectionCount();
>>>>>>>
>>>>>>> On Mon, Dec 25, 2017 at 4:16 PM, Clément Guillaume <
>>>>>>> clement@guillaume.bzh> wrote:
>>>>>>>
>>>>>>>> Here is a unit test, note that the AssertionTest table can
exit or
>>>>>>>> not
>>>>>>>> exit, assertion will fail un both case.
>>>>>>>>
>>>>>>>> > @Test
>>>>>>>> > public void test() throws IOException{
>>>>>>>> > Configuration hbaseConfig = HBaseConfiguration.create();
>>>>>>>> > String zkQuorum = "hbase123.docker";
>>>>>>>> > hbaseConfig.set(HConstants.ZOOKEEPER_QUORUM, zkQuorum);
>>>>>>>> > Connection connection = ConnectionFactory.createConnec
>>>>>>>> tion(hbaseConfig);
>>>>>>>> > Table table = connection.getTable(TableName.
>>>>>>>> valueOf("AssertionTest"));
>>>>>>>> > ResultScanner scanner = table.getScanner(new Scan());
>>>>>>>> > scanner.iterator().hasNext(); // Exception thrown here
>>>>>>>> > }
>>>>>>>> >
>>>>>>>>
>>>>>>>> PS: I'm not receiving email when a response is posted here.
I'm not
>>>>>>>> subscribed at the hbase-user mailing list but I would have
hope
>>>>>>>> that answer
>>>>>>>> get sent to me as direct email.
>>>>>>>>
>>>>>>>> 2017-12-21 12:28 GMT-08:00 Clément Guillaume <clement@guillaume.bzh
>>>>>>>> >:
>>>>>>>>
>>>>>>>> > I'm upgrading my projects from 1.2.6 to 1.4, and I have
a system
>>>>>>>> assertion
>>>>>>>> > failure (in a failsafe test that run with system assertion
enable)
>>>>>>>> >
>>>>>>>> > java.lang.AssertionError
>>>>>>>> > at org.apache.hadoop.hbase.client.ClientScanner.
>>>>>>>> > loadCache(ClientScanner.java:484) ~[hbase-shaded-client-1.4.0.ja
>>>>>>>> r:1.4.0]
>>>>>>>> > at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScan
>>>>>>>> ner.java:312)
>>>>>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>>>>>>>> > at org.apache.hadoop.hbase.client.ConnectionManager$
>>>>>>>> > HConnectionImplementation.locateRegionInMeta(ConnectionManag
>>>>>>>> er.java:1324)
>>>>>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>>>>>>>> > at org.apache.hadoop.hbase.client.ConnectionManager$
>>>>>>>> > HConnectionImplementation.locateRegion(ConnectionManager.jav
>>>>>>>> a:1221)
>>>>>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>>>>>>>> > at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutu
>>>>>>>> reImpl.
>>>>>>>> > findAllLocationsOrFail(AsyncProcess.java:1063)
>>>>>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>>>>>>>> > at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutu
>>>>>>>> reImpl.
>>>>>>>> > groupAndSendMultiAction(AsyncProcess.java:980)
>>>>>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>>>>>>>> > at org.apache.hadoop.hbase.client.AsyncProcess$
>>>>>>>> > AsyncRequestFutureImpl.access$200(AsyncProcess.java:667)
>>>>>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>>>>>>>> > at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncP
>>>>>>>> rocess.java:649)
>>>>>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>>>>>>>> > at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncP
>>>>>>>> rocess.java:612)
>>>>>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>>>>>>>> > at org.apache.hadoop.hbase.client.ConnectionManager$
>>>>>>>> > HConnectionImplementation.processBatchCallback(ConnectionMan
>>>>>>>> ager.java:2392)
>>>>>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>>>>>>>> > at org.apache.hadoop.hbase.client.HTable.batchCallback(HTable.j
>>>>>>>> ava:950)
>>>>>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>>>>>>>> >
>>>>>>>> > I have one Put action that look like that :
>>>>>>>> {"totalColumns":1,"row":"
>>>>>>>> > a7ttxdQY9euS-KEaPIMvKsRUWANCmigsaiT8_-V0Sgc\\x00visited\\x00
>>>>>>>> \\xFF\\
>>>>>>>> > xFF\\xFE\\x9F\\x85Dh\\x19","families":{"a":[{"qualifier":"
>>>>>>>> > value","vlen":17,"tag":[],"timestamp":9223372036854775807}]}}
>>>>>>>> > The cluster is a local standalone instance.
>>>>>>>> >
>>>>>>>> > I haven't dig into it yet, but it looks like the data
is
>>>>>>>> correctly written
>>>>>>>> > in hbase if I disable assertion.
>>>>>>>> > Any idea of what's causing the issue?
>>>>>>>> >
>>>>>>>> > Thank you.
>>>>>>>> >
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message