hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: assert in hbase 1.4
Date Sun, 31 Dec 2017 16:16:14 GMT
Clément:
Since you have reproduced the assertion error, can you log a JIRA ?

Thanks

On Sun, Dec 31, 2017 at 7:46 AM, Ted Yu <yuzhihong@gmail.com> wrote:

> I modified the test so that the unit test runs against branch-1.4 :
>
> +      Configuration hbaseConfig = TEST_UTIL.getConfiguration();
> +      //hbaseConfig.set(HConstants.ZOOKEEPER_QUORUM, "hbase123.docker");
>
> However the test runs to completion without assertion - meaning the
> assertion wouldn't be triggered in homogeneous cluster.
>
> Here were the last few lines in the test output:
>
> 2017-12-31 07:33:36,865 INFO  [Time-limited test] client.HBaseAdmin$CreateTableFuture(793):
> Created AssertionTest
> table created
> 2017-12-31 07:33:36,871 DEBUG [Time-limited test]
> client.ClientScanner(248): Advancing internal scanner to startKey at
> '\x01\x02\x03\x04\x05\x06\x07', inclusive
> 2017-12-31 07:33:36,873 DEBUG [hconnection-0x6e0982ab-shared--pool24-t3]
> ipc.RpcConnection(126): Use SIMPLE authentication for service
> ClientService, sasl=false
> 2017-12-31 07:33:36,873 DEBUG [hconnection-0x6e0982ab-shared--pool24-t3]
> ipc.BlockingRpcConnection(431): Connecting to
> tyus-mbp.attlocal.net/192.168.1.64:61585
> 2017-12-31 07:33:36,875 DEBUG [RpcServer.listener,port=61585]
> ipc.RpcServer$Listener(920): RpcServer.listener,port=61585: connection from
> 192.168.1.64:61633; # active          connections: 3
> 2017-12-31 07:33:36,875 INFO  [RpcServer.reader=0,bindAddress=
> tyus-mbp.attlocal.net,port=61585] ipc.RpcServer$Connection(1809): Auth
> successful for tyu (auth:SIMPLE)
> 2017-12-31 07:33:36,875 INFO  [RpcServer.reader=0,bindAddress=
> tyus-mbp.attlocal.net,port=61585] ipc.RpcServer$Connection(1839):
> Connection from 192.168.1.64 port: 61633 with   version info: version:
> "1.4.1-SNAPSHOT" url: "git://TYus-MBP.attlocal.net/Users/tyu/1.4-hbase"
> revision: "a7383851b9e62f23eb6f06eb23f09967f636c985" user: "tyu" date:
> "Sun Dec  31 07:32:58 PST 2017" src_checksum: "
> 39bdbbe40374e2094f53ce77af539671" version_major: 1 version_minor: 4
> 2017-12-31 07:33:36,917 INFO  [main] hbase.ResourceChecker(171): after:
> client.TestHCM#test Thread=325 (was 305)
>
> I assume you pointed the test at a 1.2.3 cluster which triggered the
> assertion.
>
> It seems the patch which introduced the assertion didn't take into account
> client running against older release cluster deployment.
>
> FYI
>
> On Sun, Dec 31, 2017 at 6:09 AM, Clément Guillaume <clement@guillaume.bzh>
> wrote:
>
>> I have consistent failure with this code (a table with pre-splits):
>>
>> @Test
>>> public void test() throws IOException{
>>> Configuration hbaseConfig = HBaseConfiguration.create();
>>> hbaseConfig.set(HConstants.ZOOKEEPER_QUORUM, "hbase123.docker");
>>> Connection connection = ConnectionFactory.createConnection(hbaseConfig);
>>>
>>
>>
>> TableName tableName = TableName.valueOf("AssertionTest");
>>> Admin admin = connection.getAdmin();
>>>
>>> if(!admin.tableExists(tableName)){
>>> HTableDescriptor htable = new HTableDescriptor(tableName);
>>> htable.addFamily(new HColumnDescriptor(new byte[]{(byte)'a'}));
>>> byte[][] splitPoints = {{1, 2, 3, 4, 5, 6, 7}};
>>> admin.createTable(htable, splitPoints);
>>> System.out.println("table created");
>>> }
>>>
>>> Table table = connection.getTable(tableName);
>>> ResultScanner scanner = table.getScanner(new Scan());
>>> scanner.iterator().hasNext(); // Exception thrown here
>>> }
>>
>>
>> 2017-12-30 5:04 GMT-08:00 Ted Yu <yuzhihong@gmail.com>:
>>
>>> bq. a table has been created with pre-splits
>>>
>>> Unless you record the initial number of regions somewhere, there is no
>>> built-in mechanism to provide the answer.
>>>
>>> Even if you cannot reproduce the assertion consistently, as long as the
>>> chance of reproduction is high, the unit test would still be useful.
>>>
>>> Cheers
>>>
>>> On Sat, Dec 30, 2017 at 3:31 AM, Clément Guillaume <
>>> clement@guillaume.bzh> wrote:
>>>
>>>> I dropped all the tables in my test cluster. Now I'm inconstantly able
>>>> to reproduce the issue. I'm digging more.
>>>>
>>>> I have a related question: is it possible to check if a table has been
>>>> created with pre-splits?
>>>>
>>>> I forgot to mention that I'm connecting to a version 1.2.3 cluster, but
>>>> I think this should be ok according to this post
>>>> http://mail-archives.apache.org/mod_mbox/hbase-dev/2017
>>>> 11.mbox/%3C5BAAC90F-31D8-4A5F-B9E4-BA61FF4CD40E%40gmail.com%3E
>>>>
>>>> 2017-12-25 17:10 GMT-08:00 Ted Yu <yuzhihong@gmail.com>:
>>>>
>>>>> I got the following if I use your code directly:
>>>>>
>>>>> testAssert(org.apache.hadoop.hbase.client.TestHCM)  Time elapsed:
>>>>> 0.096 sec  <<< ERROR!
>>>>> java.io.UncheckedIOException: org.apache.hadoop.hbase.TableNotFoundException:
>>>>> AssertionTest
>>>>> at org.apache.hadoop.hbase.client.TestHCM.testAssert(TestHCM.java:243)
>>>>>
>>>>> If I create the table at the beginning of the test, the test passes
>>>>> without triggering assertion.
>>>>>
>>>>> I tried both master branch and branch-1.4
>>>>>
>>>>> As I mentioned previously, the assertion in your case was probably
>>>>> related to the formation of row keys in hbase:meta table.
>>>>> See if you can reproduce the assertion by presplitting the table, etc.
>>>>>
>>>>> Here is the diff I used based on branch-1.4
>>>>>
>>>>> diff --git a/hbase-server/src/test/java/o
>>>>> rg/apache/hadoop/hbase/client/TestHCM.java
>>>>> b/hbase-server/src/test/java/org/apache/hadoop/hbase/client/
>>>>> TestHCM.java
>>>>> index a5e3a65..2944c8e 100644
>>>>> --- a/hbase-server/src/test/java/org/apache/hadoop/hbase/client/
>>>>> TestHCM.java
>>>>> +++ b/hbase-server/src/test/java/org/apache/hadoop/hbase/client/
>>>>> TestHCM.java
>>>>> @@ -246,6 +246,18 @@ public class TestHCM {
>>>>>      TEST_UTIL.shutdownMiniCluster();
>>>>>    }
>>>>>
>>>>> +  @Test
>>>>> +  public void testAssert() throws IOException{
>>>>> +    Configuration hbaseConfig = TEST_UTIL.getConfiguration();
>>>>> +    final TableName tableName = TableName.valueOf("assert");
>>>>> +    TEST_UTIL.createTable(tableName, FAM_NAM).close();
>>>>> +    String zkQuorum = "hbase123.docker";
>>>>> +    //hbaseConfig.set(HConstants.ZOOKEEPER_QUORUM, zkQuorum);
>>>>> +    Connection connection = ConnectionFactory.createConnec
>>>>> tion(hbaseConfig);
>>>>> +    Table table = connection.getTable(tableName);
>>>>> +    ResultScanner scanner = table.getScanner(new Scan());
>>>>> +    scanner.iterator().hasNext(); // Exception thrown here
>>>>> +  }
>>>>>
>>>>>    private static int getHConnectionManagerCacheSize(){
>>>>>      return HConnectionTestingUtility.getConnectionCount();
>>>>>
>>>>> On Mon, Dec 25, 2017 at 4:16 PM, Clément Guillaume <
>>>>> clement@guillaume.bzh> wrote:
>>>>>
>>>>>> Here is a unit test, note that the AssertionTest table can exit or
not
>>>>>> exit, assertion will fail un both case.
>>>>>>
>>>>>> > @Test
>>>>>> > public void test() throws IOException{
>>>>>> > Configuration hbaseConfig = HBaseConfiguration.create();
>>>>>> > String zkQuorum = "hbase123.docker";
>>>>>> > hbaseConfig.set(HConstants.ZOOKEEPER_QUORUM, zkQuorum);
>>>>>> > Connection connection = ConnectionFactory.createConnec
>>>>>> tion(hbaseConfig);
>>>>>> > Table table = connection.getTable(TableName.
>>>>>> valueOf("AssertionTest"));
>>>>>> > ResultScanner scanner = table.getScanner(new Scan());
>>>>>> > scanner.iterator().hasNext(); // Exception thrown here
>>>>>> > }
>>>>>> >
>>>>>>
>>>>>> PS: I'm not receiving email when a response is posted here. I'm not
>>>>>> subscribed at the hbase-user mailing list but I would have hope that
>>>>>> answer
>>>>>> get sent to me as direct email.
>>>>>>
>>>>>> 2017-12-21 12:28 GMT-08:00 Clément Guillaume <clement@guillaume.bzh>:
>>>>>>
>>>>>> > I'm upgrading my projects from 1.2.6 to 1.4, and I have a system
>>>>>> assertion
>>>>>> > failure (in a failsafe test that run with system assertion enable)
>>>>>> >
>>>>>> > java.lang.AssertionError
>>>>>> > at org.apache.hadoop.hbase.client.ClientScanner.
>>>>>> > loadCache(ClientScanner.java:484) ~[hbase-shaded-client-1.4.0.ja
>>>>>> r:1.4.0]
>>>>>> > at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScan
>>>>>> ner.java:312)
>>>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>>>>>> > at org.apache.hadoop.hbase.client.ConnectionManager$
>>>>>> > HConnectionImplementation.locateRegionInMeta(ConnectionManag
>>>>>> er.java:1324)
>>>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>>>>>> > at org.apache.hadoop.hbase.client.ConnectionManager$
>>>>>> > HConnectionImplementation.locateRegion(ConnectionManager.java:1221)
>>>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>>>>>> > at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutu
>>>>>> reImpl.
>>>>>> > findAllLocationsOrFail(AsyncProcess.java:1063)
>>>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>>>>>> > at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutu
>>>>>> reImpl.
>>>>>> > groupAndSendMultiAction(AsyncProcess.java:980)
>>>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>>>>>> > at org.apache.hadoop.hbase.client.AsyncProcess$
>>>>>> > AsyncRequestFutureImpl.access$200(AsyncProcess.java:667)
>>>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>>>>>> > at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncP
>>>>>> rocess.java:649)
>>>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>>>>>> > at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncP
>>>>>> rocess.java:612)
>>>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>>>>>> > at org.apache.hadoop.hbase.client.ConnectionManager$
>>>>>> > HConnectionImplementation.processBatchCallback(ConnectionMan
>>>>>> ager.java:2392)
>>>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>>>>>> > at org.apache.hadoop.hbase.client.HTable.batchCallback(HTable.j
>>>>>> ava:950)
>>>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>>>>>> >
>>>>>> > I have one Put action that look like that :
>>>>>> {"totalColumns":1,"row":"
>>>>>> > a7ttxdQY9euS-KEaPIMvKsRUWANCmigsaiT8_-V0Sgc\\x00visited\\x00\\xFF\\
>>>>>> > xFF\\xFE\\x9F\\x85Dh\\x19","families":{"a":[{"qualifier":"
>>>>>> > value","vlen":17,"tag":[],"timestamp":9223372036854775807}]}}
>>>>>> > The cluster is a local standalone instance.
>>>>>> >
>>>>>> > I haven't dig into it yet, but it looks like the data is correctly
>>>>>> written
>>>>>> > in hbase if I disable assertion.
>>>>>> > Any idea of what's causing the issue?
>>>>>> >
>>>>>> > Thank you.
>>>>>> >
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message