phoenix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Konstantinos Kougios <kostas.koug...@googlemail.com>
Subject Re: sqlline.py, timeout after 60secs, is there a way to increase the timeout period?
Date Mon, 28 Sep 2015 16:32:16 GMT
Thanks Lukas, that seems to have fixed it. I've set it both on my 
cluster and locally to my local hbase installation for it to work.

Now I am getting a different problem when doing a query:

0: jdbc:phoenix:nn.lan> select count(word),word from words group by word 
order by count(word) desc limit 10;
+------------------------------------------+------------------------------------------+
|               COUNT(WORD)                | WORD                   |
+------------------------------------------+------------------------------------------+
15/09/28 17:25:51 WARN client.ScannerCallable: Ignore, probably already 
closed
org.apache.hadoop.hbase.UnknownScannerException: 
org.apache.hadoop.hbase.UnknownScannerException: Name: 103, already closed?
     at 
org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2223)
     at 
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32205)
     at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114)
     at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)
     at 
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
     at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
     at java.lang.Thread.run(Thread.java:745)

     at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
Method)
     at 
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
     at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
     at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
     at 
org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
     at 
org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
     at 
org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:322)
     at 
org.apache.hadoop.hbase.client.ScannerCallable.close(ScannerCallable.java:357)
     at 
org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:195)
     at 
org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:142)
     at 
org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:61)
     at 
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:126)
     at 
org.apache.hadoop.hbase.client.StatsTrackingRpcRetryingCaller.callWithoutRetries(StatsTrackingRpcRetryingCaller.java:56)
     at 
org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:320)
     at 
org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:258)
     at 
org.apache.hadoop.hbase.client.ClientScanner.possiblyNextScanner(ClientScanner.java:241)
     at 
org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:532)
     at 
org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:364)
     at 
org.apache.phoenix.iterate.ScanningResultIterator.next(ScanningResultIterator.java:55)
     at 
org.apache.phoenix.iterate.TableResultIterator.next(TableResultIterator.java:107)
     at 
org.apache.phoenix.iterate.OrderedResultIterator.getResultIterator(OrderedResultIterator.java:240)
     at 
org.apache.phoenix.iterate.OrderedResultIterator.peek(OrderedResultIterator.java:263)
     at 
org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:112)
     at 
org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:100)
     at java.util.concurrent.FutureTask.run(FutureTask.java:266)
     at 
org.apache.phoenix.job.JobManager$InstrumentedJobFutureTask.run(JobManager.java:183)
     at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
     at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
     at java.lang.Thread.run(Thread.java:745)
Caused by: 
org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.UnknownScannerException):

org.apache.hadoop.hbase.UnknownScannerException: Name: 103, already closed?
     at 
org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2223)
     at 
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32205)
     at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114)
     at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)
     at 
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
     at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
     at java.lang.Thread.run(Thread.java:745)

     at 
org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1196)
     at 
org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:213)
     at 
org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:287)
     at 
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.scan(ClientProtos.java:32651)
     at 
org.apache.hadoop.hbase.client.ScannerCallable.close(ScannerCallable.java:355)
     ... 21 more

Do you know if it is related to phoenix.query.keepAliveMs ? (currently 
my cluster is busy but will give it a try later on)



On 28/09/15 15:49, Lukáš Lalinský wrote:
> You need to set "hbase.rpc.timeout" to the same value as you have for 
> "phoenix.query.timeoutMs".
>
> It seems that in the pre-Apache version of Phoenix it was set 
> automatically:
>
> https://issues.apache.org/jira/browse/PHOENIX-269?focusedCommentId=14681924&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14681924
>
> Lukas
>
>
> On Mon, Sep 28, 2015 at 4:41 PM, Konstantinos Kougios 
> <kostas.kougios@googlemail.com <mailto:kostas.kougios@googlemail.com>> 
> wrote:
>
>     I have a fairly big table on a noy-so-fairly-powerfull cluster, so
>     it takes a lot of time for queries to respond. I don't mind that
>     but it times out for many queries :
>
>     0: jdbc:phoenix:nn.lan> select count(*) from words;
>     +------------------------------------------+
>     |                 COUNT(1)                 |
>     +------------------------------------------+
>     java.lang.RuntimeException:
>     org.apache.phoenix.exception.PhoenixIOException:
>     org.apache.phoenix.exception.PhoenixIOException: Failed after
>     attempts=1, exceptions:
>     Mon Sep 28 15:32:25 BST 2015,
>     RpcRetryingCaller{globalStartTime=1443450685716, pause=100,
>     retries=1},
>     org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed
>     after attempts=1, exceptions:
>     Mon Sep 28 15:32:25 BST 2015,
>     RpcRetryingCaller{globalStartTime=1443450685716, pause=100,
>     retries=1},
>     org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed
>     after attempts=1, exceptions:
>     Mon Sep 28 15:32:25 BST 2015,
>     RpcRetryingCaller{globalStartTime=1443450685716, pause=100,
>     retries=1}, java.io.IOException: Call to d2.lan/192.168.0.30:16020
>     <http://192.168.0.30:16020> failed on local exception:
>     org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=842,
>     waitTime=60001, operationTimeout=60000 expired.
>
>
>
>         at sqlline.IncrementalRows.hasNext(IncrementalRows.java:73)
>         at sqlline.TableOutputFormat.print(TableOutputFormat.java:33)
>         at sqlline.SqlLine.print(SqlLine.java:1653)
>         at sqlline.Commands.execute(Commands.java:833)
>         at sqlline.Commands.sql(Commands.java:732)
>         at sqlline.SqlLine.dispatch(SqlLine.java:808)
>         at sqlline.SqlLine.begin(SqlLine.java:681)
>         at sqlline.SqlLine.start(SqlLine.java:398)
>         at sqlline.SqlLine.main(SqlLine.java:292)
>     0: jdbc:phoenix:nn.lan> Closing:
>     org.apache.phoenix.jdbc.PhoenixConnection
>
>
>     I did try some settings on hbase-site without luck:
>
>
>     phoenix.query.timeoutMs =6000000 hbase.client.operation.timeout =1200000 hbase.client.backpressure.enabled
=true hbase.client.retries.number =1
>
>     Any ideas how can this be fixed?
>
>     It seems the problem is that the timeout for iterating through
>     results is 60secs. I assume if it doesn't get 1 result within that
>     period, it times out. Since this is a count(*) query with only 1
>     row, it does timeout.
>
>     Thanks
>
>


Mime
View raw message