phoenix-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Purtell (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (PHOENIX-3702) RenewLeaseIT#testLeaseDoesNotTimeout failing on 4.x-HBase-1.1 and master branches
Date Wed, 01 Mar 2017 23:21:45 GMT

    [ https://issues.apache.org/jira/browse/PHOENIX-3702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15891274#comment-15891274
] 

Andrew Purtell commented on PHOENIX-3702:
-----------------------------------------

Copying over my comments from HBASE-17714

[~samarthjain] Are you sure RenewLeaseIT actually renews the lease or allows for a client
heartbeat to happen before the RPC times out? The test sets a very short RPC timeout (2000ms)
but makes no other configuration changes

The release notes on HBASE-13090 say

{quote}
To ensure that timeout checks do not occur too often (which would hurt the performance of
scans), the configuration "hbase.cells.scanned.per.heartbeat.check" has been introduced. This
configuration controls how often System.currentTimeMillis() is called to update the progress
towards the time limit. Currently, the default value of this configuration value is 10000.
{quote}


> RenewLeaseIT#testLeaseDoesNotTimeout failing on 4.x-HBase-1.1 and master branches
> ---------------------------------------------------------------------------------
>
>                 Key: PHOENIX-3702
>                 URL: https://issues.apache.org/jira/browse/PHOENIX-3702
>             Project: Phoenix
>          Issue Type: Bug
>            Reporter: Samarth Jain
>            Assignee: Samarth Jain
>            Priority: Blocker
>             Fix For: 4.10.0
>
>
> Failure stacktrace:
> {code}
> org.apache.phoenix.exception.PhoenixIOException: org.apache.phoenix.exception.PhoenixIOException:
Failed after retry of OutOfOrderScannerNextException: was there a rpc timeout?
> 	at org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:113)
> 	at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:852)
> 	at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:796)
> 	at org.apache.phoenix.iterate.ConcatResultIterator.getIterators(ConcatResultIterator.java:50)
> 	at org.apache.phoenix.iterate.ConcatResultIterator.currentIterator(ConcatResultIterator.java:97)
> 	at org.apache.phoenix.iterate.ConcatResultIterator.next(ConcatResultIterator.java:117)
> 	at org.apache.phoenix.iterate.BaseGroupedAggregatingResultIterator.next(BaseGroupedAggregatingResultIterator.java:64)
> 	at org.apache.phoenix.iterate.UngroupedAggregatingResultIterator.next(UngroupedAggregatingResultIterator.java:39)
> 	at org.apache.phoenix.jdbc.PhoenixResultSet.next(PhoenixResultSet.java:778)
> 	at org.apache.phoenix.end2end.RenewLeaseIT.testLeaseDoesNotTimeout(RenewLeaseIT.java:68)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:606)
> 	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> 	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> 	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> 	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> 	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> 	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> 	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> 	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> 	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> 	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> 	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> 	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> 	at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> 	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> 	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> 	at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
> 	at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
> 	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
> 	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
> 	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
> 	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
> Caused by: java.util.concurrent.ExecutionException: org.apache.phoenix.exception.PhoenixIOException:
Failed after retry of OutOfOrderScannerNextException: was there a rpc timeout?
> 	at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> 	at java.util.concurrent.FutureTask.get(FutureTask.java:202)
> 	at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:847)
> 	... 35 more
> Caused by: org.apache.phoenix.exception.PhoenixIOException: Failed after retry of OutOfOrderScannerNextException:
was there a rpc timeout?
> 	at org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:113)
> 	at org.apache.phoenix.iterate.TableResultIterator.next(TableResultIterator.java:133)
> 	at org.apache.phoenix.iterate.LookAheadResultIterator$1.advance(LookAheadResultIterator.java:47)
> 	at org.apache.phoenix.iterate.LookAheadResultIterator.init(LookAheadResultIterator.java:59)
> 	at org.apache.phoenix.iterate.LookAheadResultIterator.peek(LookAheadResultIterator.java:73)
> 	at org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:121)
> 	at org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:1)
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> 	at org.apache.phoenix.job.JobManager$InstrumentedJobFutureTask.run(JobManager.java:183)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> 	at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.phoenix.exception.PhoenixIOException: Failed after retry of OutOfOrderScannerNextException:
was there a rpc timeout?
> 	at org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:113)
> 	at org.apache.phoenix.iterate.ScanningResultIterator.next(ScanningResultIterator.java:65)
> 	at org.apache.phoenix.iterate.TableResultIterator.next(TableResultIterator.java:126)
> 	... 10 more
> Caused by: org.apache.hadoop.hbase.DoNotRetryIOException: Failed after retry of OutOfOrderScannerNextException:
was there a rpc timeout?
> 	at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:483)
> 	at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:371)
> 	at org.apache.phoenix.iterate.ScanningResultIterator.next(ScanningResultIterator.java:55)
> 	... 11 more
> Caused by: org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException: org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException:
Expected nextCallSeq: 1 But the nextCallSeq got from client: 0; request=scanner_id: 39 number_of_rows:
2147483647 close_scanner: false next_call_seq: 0 client_handles_partials: true client_handles_heartbeats:
true renew: false
> 	at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2351)
> 	at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32385)
> 	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2137)
> 	at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:107)
> 	at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
> 	at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
> 	at java.lang.Thread.run(Thread.java:745)
> 	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> 	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> 	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> 	at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> 	at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
> 	at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
> 	at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:326)
> 	at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:257)
> 	at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:63)
> 	at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:210)
> 	at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:364)
> 	at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:338)
> 	at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:136)
> 	at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:65)
> 	... 3 more
> Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException):
org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException: Expected nextCallSeq: 1
But the nextCallSeq got from client: 0; request=scanner_id: 39 number_of_rows: 2147483647
close_scanner: false next_call_seq: 0 client_handles_partials: true client_handles_heartbeats:
true renew: false
> 	at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2351)
> 	at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32385)
> 	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2137)
> 	at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:107)
> 	at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
> 	at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
> 	at java.lang.Thread.run(Thread.java:745)
> 	at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1234)
> 	at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:223)
> 	at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:328)
> 	at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.scan(ClientProtos.java:32831)
> 	at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:215)
> 	... 9 more
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message