hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: assert in hbase 1.4
Date Sun, 31 Dec 2017 15:46:45 GMT
I modified the test so that the unit test runs against branch-1.4 :

+      Configuration hbaseConfig = TEST_UTIL.getConfiguration();
+      //hbaseConfig.set(HConstants.ZOOKEEPER_QUORUM, "hbase123.docker");

However the test runs to completion without assertion - meaning the
assertion wouldn't be triggered in homogeneous cluster.

Here were the last few lines in the test output:

2017-12-31 07:33:36,865 INFO  [Time-limited test]
client.HBaseAdmin$CreateTableFuture(793): Created AssertionTest
table created
2017-12-31 07:33:36,871 DEBUG [Time-limited test]
client.ClientScanner(248): Advancing internal scanner to startKey at
'\x01\x02\x03\x04\x05\x06\x07', inclusive
2017-12-31 07:33:36,873 DEBUG [hconnection-0x6e0982ab-shared--pool24-t3]
ipc.RpcConnection(126): Use SIMPLE authentication for service
ClientService, sasl=false
2017-12-31 07:33:36,873 DEBUG [hconnection-0x6e0982ab-shared--pool24-t3]
ipc.BlockingRpcConnection(431): Connecting to
tyus-mbp.attlocal.net/192.168.1.64:61585
2017-12-31 07:33:36,875 DEBUG [RpcServer.listener,port=61585]
ipc.RpcServer$Listener(920): RpcServer.listener,port=61585: connection from
192.168.1.64:61633; # active          connections: 3
2017-12-31 07:33:36,875 INFO  [RpcServer.reader=0,bindAddress=
tyus-mbp.attlocal.net,port=61585] ipc.RpcServer$Connection(1809): Auth
successful for tyu (auth:SIMPLE)
2017-12-31 07:33:36,875 INFO  [RpcServer.reader=0,bindAddress=
tyus-mbp.attlocal.net,port=61585] ipc.RpcServer$Connection(1839):
Connection from 192.168.1.64 port: 61633 with   version info: version:
"1.4.1-SNAPSHOT" url: "git://TYus-MBP.attlocal.net/Users/tyu/1.4-hbase"
revision: "a7383851b9e62f23eb6f06eb23f09967f636c985" user: "tyu" date: "Sun
Dec  31 07:32:58 PST 2017" src_checksum: "39bdbbe40374e2094f53ce77af539671"
version_major: 1 version_minor: 4
2017-12-31 07:33:36,917 INFO  [main] hbase.ResourceChecker(171): after:
client.TestHCM#test Thread=325 (was 305)

I assume you pointed the test at a 1.2.3 cluster which triggered the
assertion.

It seems the patch which introduced the assertion didn't take into account
client running against older release cluster deployment.

FYI

On Sun, Dec 31, 2017 at 6:09 AM, Clément Guillaume <clement@guillaume.bzh>
wrote:

> I have consistent failure with this code (a table with pre-splits):
>
> @Test
>> public void test() throws IOException{
>> Configuration hbaseConfig = HBaseConfiguration.create();
>> hbaseConfig.set(HConstants.ZOOKEEPER_QUORUM, "hbase123.docker");
>> Connection connection = ConnectionFactory.createConnection(hbaseConfig);
>>
>
>
> TableName tableName = TableName.valueOf("AssertionTest");
>> Admin admin = connection.getAdmin();
>>
>> if(!admin.tableExists(tableName)){
>> HTableDescriptor htable = new HTableDescriptor(tableName);
>> htable.addFamily(new HColumnDescriptor(new byte[]{(byte)'a'}));
>> byte[][] splitPoints = {{1, 2, 3, 4, 5, 6, 7}};
>> admin.createTable(htable, splitPoints);
>> System.out.println("table created");
>> }
>>
>> Table table = connection.getTable(tableName);
>> ResultScanner scanner = table.getScanner(new Scan());
>> scanner.iterator().hasNext(); // Exception thrown here
>> }
>
>
> 2017-12-30 5:04 GMT-08:00 Ted Yu <yuzhihong@gmail.com>:
>
>> bq. a table has been created with pre-splits
>>
>> Unless you record the initial number of regions somewhere, there is no
>> built-in mechanism to provide the answer.
>>
>> Even if you cannot reproduce the assertion consistently, as long as the
>> chance of reproduction is high, the unit test would still be useful.
>>
>> Cheers
>>
>> On Sat, Dec 30, 2017 at 3:31 AM, Clément Guillaume <clement@guillaume.bzh
>> > wrote:
>>
>>> I dropped all the tables in my test cluster. Now I'm inconstantly able
>>> to reproduce the issue. I'm digging more.
>>>
>>> I have a related question: is it possible to check if a table has been
>>> created with pre-splits?
>>>
>>> I forgot to mention that I'm connecting to a version 1.2.3 cluster, but
>>> I think this should be ok according to this post
>>> http://mail-archives.apache.org/mod_mbox/hbase-dev/2017
>>> 11.mbox/%3C5BAAC90F-31D8-4A5F-B9E4-BA61FF4CD40E%40gmail.com%3E
>>>
>>> 2017-12-25 17:10 GMT-08:00 Ted Yu <yuzhihong@gmail.com>:
>>>
>>>> I got the following if I use your code directly:
>>>>
>>>> testAssert(org.apache.hadoop.hbase.client.TestHCM)  Time elapsed:
>>>> 0.096 sec  <<< ERROR!
>>>> java.io.UncheckedIOException: org.apache.hadoop.hbase.TableNotFoundException:
>>>> AssertionTest
>>>> at org.apache.hadoop.hbase.client.TestHCM.testAssert(TestHCM.java:243)
>>>>
>>>> If I create the table at the beginning of the test, the test passes
>>>> without triggering assertion.
>>>>
>>>> I tried both master branch and branch-1.4
>>>>
>>>> As I mentioned previously, the assertion in your case was probably
>>>> related to the formation of row keys in hbase:meta table.
>>>> See if you can reproduce the assertion by presplitting the table, etc.
>>>>
>>>> Here is the diff I used based on branch-1.4
>>>>
>>>> diff --git a/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestHCM.java
>>>> b/hbase-server/src/test/java/org/apache/hadoop/hbase/client/
>>>> TestHCM.java
>>>> index a5e3a65..2944c8e 100644
>>>> --- a/hbase-server/src/test/java/org/apache/hadoop/hbase/client/
>>>> TestHCM.java
>>>> +++ b/hbase-server/src/test/java/org/apache/hadoop/hbase/client/
>>>> TestHCM.java
>>>> @@ -246,6 +246,18 @@ public class TestHCM {
>>>>      TEST_UTIL.shutdownMiniCluster();
>>>>    }
>>>>
>>>> +  @Test
>>>> +  public void testAssert() throws IOException{
>>>> +    Configuration hbaseConfig = TEST_UTIL.getConfiguration();
>>>> +    final TableName tableName = TableName.valueOf("assert");
>>>> +    TEST_UTIL.createTable(tableName, FAM_NAM).close();
>>>> +    String zkQuorum = "hbase123.docker";
>>>> +    //hbaseConfig.set(HConstants.ZOOKEEPER_QUORUM, zkQuorum);
>>>> +    Connection connection = ConnectionFactory.createConnec
>>>> tion(hbaseConfig);
>>>> +    Table table = connection.getTable(tableName);
>>>> +    ResultScanner scanner = table.getScanner(new Scan());
>>>> +    scanner.iterator().hasNext(); // Exception thrown here
>>>> +  }
>>>>
>>>>    private static int getHConnectionManagerCacheSize(){
>>>>      return HConnectionTestingUtility.getConnectionCount();
>>>>
>>>> On Mon, Dec 25, 2017 at 4:16 PM, Clément Guillaume <
>>>> clement@guillaume.bzh> wrote:
>>>>
>>>>> Here is a unit test, note that the AssertionTest table can exit or not
>>>>> exit, assertion will fail un both case.
>>>>>
>>>>> > @Test
>>>>> > public void test() throws IOException{
>>>>> > Configuration hbaseConfig = HBaseConfiguration.create();
>>>>> > String zkQuorum = "hbase123.docker";
>>>>> > hbaseConfig.set(HConstants.ZOOKEEPER_QUORUM, zkQuorum);
>>>>> > Connection connection = ConnectionFactory.createConnec
>>>>> tion(hbaseConfig);
>>>>> > Table table = connection.getTable(TableName.
>>>>> valueOf("AssertionTest"));
>>>>> > ResultScanner scanner = table.getScanner(new Scan());
>>>>> > scanner.iterator().hasNext(); // Exception thrown here
>>>>> > }
>>>>> >
>>>>>
>>>>> PS: I'm not receiving email when a response is posted here. I'm not
>>>>> subscribed at the hbase-user mailing list but I would have hope that
>>>>> answer
>>>>> get sent to me as direct email.
>>>>>
>>>>> 2017-12-21 12:28 GMT-08:00 Clément Guillaume <clement@guillaume.bzh>:
>>>>>
>>>>> > I'm upgrading my projects from 1.2.6 to 1.4, and I have a system
>>>>> assertion
>>>>> > failure (in a failsafe test that run with system assertion enable)
>>>>> >
>>>>> > java.lang.AssertionError
>>>>> > at org.apache.hadoop.hbase.client.ClientScanner.
>>>>> > loadCache(ClientScanner.java:484) ~[hbase-shaded-client-1.4.0.ja
>>>>> r:1.4.0]
>>>>> > at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScan
>>>>> ner.java:312)
>>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>>>>> > at org.apache.hadoop.hbase.client.ConnectionManager$
>>>>> > HConnectionImplementation.locateRegionInMeta(ConnectionManag
>>>>> er.java:1324)
>>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>>>>> > at org.apache.hadoop.hbase.client.ConnectionManager$
>>>>> > HConnectionImplementation.locateRegion(ConnectionManager.java:1221)
>>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>>>>> > at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutu
>>>>> reImpl.
>>>>> > findAllLocationsOrFail(AsyncProcess.java:1063)
>>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>>>>> > at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutu
>>>>> reImpl.
>>>>> > groupAndSendMultiAction(AsyncProcess.java:980)
>>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>>>>> > at org.apache.hadoop.hbase.client.AsyncProcess$
>>>>> > AsyncRequestFutureImpl.access$200(AsyncProcess.java:667)
>>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>>>>> > at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncP
>>>>> rocess.java:649)
>>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>>>>> > at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncP
>>>>> rocess.java:612)
>>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>>>>> > at org.apache.hadoop.hbase.client.ConnectionManager$
>>>>> > HConnectionImplementation.processBatchCallback(ConnectionMan
>>>>> ager.java:2392)
>>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>>>>> > at org.apache.hadoop.hbase.client.HTable.batchCallback(HTable.j
>>>>> ava:950)
>>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>>>>> >
>>>>> > I have one Put action that look like that : {"totalColumns":1,"row":"
>>>>> > a7ttxdQY9euS-KEaPIMvKsRUWANCmigsaiT8_-V0Sgc\\x00visited\\x00\\xFF\\
>>>>> > xFF\\xFE\\x9F\\x85Dh\\x19","families":{"a":[{"qualifier":"
>>>>> > value","vlen":17,"tag":[],"timestamp":9223372036854775807}]}}
>>>>> > The cluster is a local standalone instance.
>>>>> >
>>>>> > I haven't dig into it yet, but it looks like the data is correctly
>>>>> written
>>>>> > in hbase if I disable assertion.
>>>>> > Any idea of what's causing the issue?
>>>>> >
>>>>> > Thank you.
>>>>> >
>>>>>
>>>>
>>>>
>>>
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message