hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Clément Guillaume <clem...@guillaume.bzh>
Subject Re: assert in hbase 1.4
Date Wed, 03 Jan 2018 22:54:49 GMT
Done https://issues.apache.org/jira/browse/HBASE-19700

2017-12-31 8:16 GMT-08:00 Ted Yu <yuzhihong@gmail.com>:

> Clément:
> Since you have reproduced the assertion error, can you log a JIRA ?
>
> Thanks
>
> On Sun, Dec 31, 2017 at 7:46 AM, Ted Yu <yuzhihong@gmail.com> wrote:
>
>> I modified the test so that the unit test runs against branch-1.4 :
>>
>> +      Configuration hbaseConfig = TEST_UTIL.getConfiguration();
>> +      //hbaseConfig.set(HConstants.ZOOKEEPER_QUORUM, "hbase123.docker");
>>
>> However the test runs to completion without assertion - meaning the
>> assertion wouldn't be triggered in homogeneous cluster.
>>
>> Here were the last few lines in the test output:
>>
>> 2017-12-31 07:33:36,865 INFO  [Time-limited test]
>> client.HBaseAdmin$CreateTableFuture(793): Created AssertionTest
>> table created
>> 2017-12-31 07:33:36,871 DEBUG [Time-limited test]
>> client.ClientScanner(248): Advancing internal scanner to startKey at
>> '\x01\x02\x03\x04\x05\x06\x07', inclusive
>> 2017-12-31 07:33:36,873 DEBUG [hconnection-0x6e0982ab-shared--pool24-t3]
>> ipc.RpcConnection(126): Use SIMPLE authentication for service
>> ClientService, sasl=false
>> 2017-12-31 07:33:36,873 DEBUG [hconnection-0x6e0982ab-shared--pool24-t3]
>> ipc.BlockingRpcConnection(431): Connecting to
>> tyus-mbp.attlocal.net/192.168.1.64:61585
>> 2017-12-31 07:33:36,875 DEBUG [RpcServer.listener,port=61585]
>> ipc.RpcServer$Listener(920): RpcServer.listener,port=61585: connection from
>> 192.168.1.64:61633; # active          connections: 3
>> 2017-12-31 07:33:36,875 INFO  [RpcServer.reader=0,bindAddress=
>> tyus-mbp.attlocal.net,port=61585] ipc.RpcServer$Connection(1809): Auth
>> successful for tyu (auth:SIMPLE)
>> 2017-12-31 07:33:36,875 INFO  [RpcServer.reader=0,bindAddress=
>> tyus-mbp.attlocal.net,port=61585] ipc.RpcServer$Connection(1839):
>> Connection from 192.168.1.64 port: 61633 with   version info: version:
>> "1.4.1-SNAPSHOT" url: "git://TYus-MBP.attlocal.net/Users/tyu/1.4-hbase"
>> revision: "a7383851b9e62f23eb6f06eb23f09967f636c985" user: "tyu" date:
>> "Sun Dec  31 07:32:58 PST 2017" src_checksum: "39bdbbe40374e2094f53ce77af539671"
>> version_major: 1 version_minor: 4
>> 2017-12-31 07:33:36,917 INFO  [main] hbase.ResourceChecker(171): after:
>> client.TestHCM#test Thread=325 (was 305)
>>
>> I assume you pointed the test at a 1.2.3 cluster which triggered the
>> assertion.
>>
>> It seems the patch which introduced the assertion didn't take into
>> account client running against older release cluster deployment.
>>
>> FYI
>>
>> On Sun, Dec 31, 2017 at 6:09 AM, Clément Guillaume <clement@guillaume.bzh
>> > wrote:
>>
>>> I have consistent failure with this code (a table with pre-splits):
>>>
>>> @Test
>>>> public void test() throws IOException{
>>>> Configuration hbaseConfig = HBaseConfiguration.create();
>>>> hbaseConfig.set(HConstants.ZOOKEEPER_QUORUM, "hbase123.docker");
>>>> Connection connection = ConnectionFactory.createConnec
>>>> tion(hbaseConfig);
>>>>
>>>
>>>
>>> TableName tableName = TableName.valueOf("AssertionTest");
>>>> Admin admin = connection.getAdmin();
>>>>
>>>> if(!admin.tableExists(tableName)){
>>>> HTableDescriptor htable = new HTableDescriptor(tableName);
>>>> htable.addFamily(new HColumnDescriptor(new byte[]{(byte)'a'}));
>>>> byte[][] splitPoints = {{1, 2, 3, 4, 5, 6, 7}};
>>>> admin.createTable(htable, splitPoints);
>>>> System.out.println("table created");
>>>> }
>>>>
>>>> Table table = connection.getTable(tableName);
>>>> ResultScanner scanner = table.getScanner(new Scan());
>>>> scanner.iterator().hasNext(); // Exception thrown here
>>>> }
>>>
>>>
>>> 2017-12-30 5:04 GMT-08:00 Ted Yu <yuzhihong@gmail.com>:
>>>
>>>> bq. a table has been created with pre-splits
>>>>
>>>> Unless you record the initial number of regions somewhere, there is no
>>>> built-in mechanism to provide the answer.
>>>>
>>>> Even if you cannot reproduce the assertion consistently, as long as the
>>>> chance of reproduction is high, the unit test would still be useful.
>>>>
>>>> Cheers
>>>>
>>>> On Sat, Dec 30, 2017 at 3:31 AM, Clément Guillaume <
>>>> clement@guillaume.bzh> wrote:
>>>>
>>>>> I dropped all the tables in my test cluster. Now I'm inconstantly able
>>>>> to reproduce the issue. I'm digging more.
>>>>>
>>>>> I have a related question: is it possible to check if a table has been
>>>>> created with pre-splits?
>>>>>
>>>>> I forgot to mention that I'm connecting to a version 1.2.3 cluster,
>>>>> but I think this should be ok according to this post
>>>>> http://mail-archives.apache.org/mod_mbox/hbase-dev/2017
>>>>> 11.mbox/%3C5BAAC90F-31D8-4A5F-B9E4-BA61FF4CD40E%40gmail.com%3E
>>>>>
>>>>> 2017-12-25 17:10 GMT-08:00 Ted Yu <yuzhihong@gmail.com>:
>>>>>
>>>>>> I got the following if I use your code directly:
>>>>>>
>>>>>> testAssert(org.apache.hadoop.hbase.client.TestHCM)  Time elapsed:
>>>>>> 0.096 sec  <<< ERROR!
>>>>>> java.io.UncheckedIOException: org.apache.hadoop.hbase.TableNotFoundException:
>>>>>> AssertionTest
>>>>>> at org.apache.hadoop.hbase.client.TestHCM.testAssert(TestHCM.ja
>>>>>> va:243)
>>>>>>
>>>>>> If I create the table at the beginning of the test, the test passes
>>>>>> without triggering assertion.
>>>>>>
>>>>>> I tried both master branch and branch-1.4
>>>>>>
>>>>>> As I mentioned previously, the assertion in your case was probably
>>>>>> related to the formation of row keys in hbase:meta table.
>>>>>> See if you can reproduce the assertion by presplitting the table,
etc.
>>>>>>
>>>>>> Here is the diff I used based on branch-1.4
>>>>>>
>>>>>> diff --git a/hbase-server/src/test/java/o
>>>>>> rg/apache/hadoop/hbase/client/TestHCM.java
>>>>>> b/hbase-server/src/test/java/org/apache/hadoop/hbase/client/
>>>>>> TestHCM.java
>>>>>> index a5e3a65..2944c8e 100644
>>>>>> --- a/hbase-server/src/test/java/org/apache/hadoop/hbase/client/
>>>>>> TestHCM.java
>>>>>> +++ b/hbase-server/src/test/java/org/apache/hadoop/hbase/client/
>>>>>> TestHCM.java
>>>>>> @@ -246,6 +246,18 @@ public class TestHCM {
>>>>>>      TEST_UTIL.shutdownMiniCluster();
>>>>>>    }
>>>>>>
>>>>>> +  @Test
>>>>>> +  public void testAssert() throws IOException{
>>>>>> +    Configuration hbaseConfig = TEST_UTIL.getConfiguration();
>>>>>> +    final TableName tableName = TableName.valueOf("assert");
>>>>>> +    TEST_UTIL.createTable(tableName, FAM_NAM).close();
>>>>>> +    String zkQuorum = "hbase123.docker";
>>>>>> +    //hbaseConfig.set(HConstants.ZOOKEEPER_QUORUM, zkQuorum);
>>>>>> +    Connection connection = ConnectionFactory.createConnec
>>>>>> tion(hbaseConfig);
>>>>>> +    Table table = connection.getTable(tableName);
>>>>>> +    ResultScanner scanner = table.getScanner(new Scan());
>>>>>> +    scanner.iterator().hasNext(); // Exception thrown here
>>>>>> +  }
>>>>>>
>>>>>>    private static int getHConnectionManagerCacheSize(){
>>>>>>      return HConnectionTestingUtility.getConnectionCount();
>>>>>>
>>>>>> On Mon, Dec 25, 2017 at 4:16 PM, Clément Guillaume <
>>>>>> clement@guillaume.bzh> wrote:
>>>>>>
>>>>>>> Here is a unit test, note that the AssertionTest table can exit
or
>>>>>>> not
>>>>>>> exit, assertion will fail un both case.
>>>>>>>
>>>>>>> > @Test
>>>>>>> > public void test() throws IOException{
>>>>>>> > Configuration hbaseConfig = HBaseConfiguration.create();
>>>>>>> > String zkQuorum = "hbase123.docker";
>>>>>>> > hbaseConfig.set(HConstants.ZOOKEEPER_QUORUM, zkQuorum);
>>>>>>> > Connection connection = ConnectionFactory.createConnec
>>>>>>> tion(hbaseConfig);
>>>>>>> > Table table = connection.getTable(TableName.
>>>>>>> valueOf("AssertionTest"));
>>>>>>> > ResultScanner scanner = table.getScanner(new Scan());
>>>>>>> > scanner.iterator().hasNext(); // Exception thrown here
>>>>>>> > }
>>>>>>> >
>>>>>>>
>>>>>>> PS: I'm not receiving email when a response is posted here. I'm
not
>>>>>>> subscribed at the hbase-user mailing list but I would have hope
that
>>>>>>> answer
>>>>>>> get sent to me as direct email.
>>>>>>>
>>>>>>> 2017-12-21 12:28 GMT-08:00 Clément Guillaume <clement@guillaume.bzh
>>>>>>> >:
>>>>>>>
>>>>>>> > I'm upgrading my projects from 1.2.6 to 1.4, and I have
a system
>>>>>>> assertion
>>>>>>> > failure (in a failsafe test that run with system assertion
enable)
>>>>>>> >
>>>>>>> > java.lang.AssertionError
>>>>>>> > at org.apache.hadoop.hbase.client.ClientScanner.
>>>>>>> > loadCache(ClientScanner.java:484) ~[hbase-shaded-client-1.4.0.ja
>>>>>>> r:1.4.0]
>>>>>>> > at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScan
>>>>>>> ner.java:312)
>>>>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>>>>>>> > at org.apache.hadoop.hbase.client.ConnectionManager$
>>>>>>> > HConnectionImplementation.locateRegionInMeta(ConnectionManag
>>>>>>> er.java:1324)
>>>>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>>>>>>> > at org.apache.hadoop.hbase.client.ConnectionManager$
>>>>>>> > HConnectionImplementation.locateRegion(ConnectionManager.jav
>>>>>>> a:1221)
>>>>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>>>>>>> > at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutu
>>>>>>> reImpl.
>>>>>>> > findAllLocationsOrFail(AsyncProcess.java:1063)
>>>>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>>>>>>> > at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutu
>>>>>>> reImpl.
>>>>>>> > groupAndSendMultiAction(AsyncProcess.java:980)
>>>>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>>>>>>> > at org.apache.hadoop.hbase.client.AsyncProcess$
>>>>>>> > AsyncRequestFutureImpl.access$200(AsyncProcess.java:667)
>>>>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>>>>>>> > at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncP
>>>>>>> rocess.java:649)
>>>>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>>>>>>> > at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncP
>>>>>>> rocess.java:612)
>>>>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>>>>>>> > at org.apache.hadoop.hbase.client.ConnectionManager$
>>>>>>> > HConnectionImplementation.processBatchCallback(ConnectionMan
>>>>>>> ager.java:2392)
>>>>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>>>>>>> > at org.apache.hadoop.hbase.client.HTable.batchCallback(HTable.j
>>>>>>> ava:950)
>>>>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>>>>>>> >
>>>>>>> > I have one Put action that look like that :
>>>>>>> {"totalColumns":1,"row":"
>>>>>>> > a7ttxdQY9euS-KEaPIMvKsRUWANCmigsaiT8_-V0Sgc\\x00visited\\x00
>>>>>>> \\xFF\\
>>>>>>> > xFF\\xFE\\x9F\\x85Dh\\x19","families":{"a":[{"qualifier":"
>>>>>>> > value","vlen":17,"tag":[],"timestamp":9223372036854775807}]}}
>>>>>>> > The cluster is a local standalone instance.
>>>>>>> >
>>>>>>> > I haven't dig into it yet, but it looks like the data is
correctly
>>>>>>> written
>>>>>>> > in hbase if I disable assertion.
>>>>>>> > Any idea of what's causing the issue?
>>>>>>> >
>>>>>>> > Thank you.
>>>>>>> >
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message