storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rahul R <rahul8...@gmail.com>
Subject Re: [Storm v0.9.6] DRPC Server periodically crashes after sometime
Date Sat, 13 Aug 2016 06:35:29 GMT
After increasing memory,  Most of the drpc request fail.  I get the
following error,

===
2016-08-12T23:50:12.461+0000 b.s.d.drpc [WARN] Timeout DRPC request id: 10
start at 1471045208
2016-08-12T23:55:17.471+0000 b.s.d.drpc [WARN] Timeout DRPC request id: 11
start at 1471045515
2016-08-12T23:55:32.473+0000 b.s.d.drpc [WARN] Timeout DRPC request id: 12
start at 1471045530
2016-08-12T23:55:47.473+0000 b.s.d.drpc [WARN] Timeout DRPC request id: 13
start at 1471045545
2016-08-12T23:56:27.475+0000 b.s.d.drpc [WARN] Timeout DRPC request id: 14
start at 1471045586
2016-08-12T23:59:37.482+0000 b.s.d.drpc [WARN] Timeout DRPC request id: 15
start at 1471045775
2016-08-13T00:01:12.486+0000 b.s.d.drpc [WARN] Timeout DRPC request id: 16
start at 1471045867
2016-08-13T00:09:42.501+0000 b.s.d.drpc [WARN] Timeout DRPC request id: 17
start at 1471046381
===

Is there anyway to debug drpc based issues ? Sometime restarting drpc
server resolves the issue, sometimes, it doesnt.


Thanks,
./Rahul





On Wed, Aug 10, 2016 at 11:17 PM, Rahul R <rahul8590@gmail.com> wrote:

> Sure , thats one of the solutions. But I am not sure, whats causing DRPC
> to consume soo much of memory. Any suggestions of how to debug this ?
>
> Thanks,
> ./Rahul
>
> On Wed, Aug 10, 2016 at 6:24 PM, darion.yaphet <flykos@163.com> wrote:
>
>> Maybe you should check the JVM memory parameter -Xms and -Xmx to found
>> memory usage.
>>
>>
>>
>> At 2016-08-11 08:00:54, "Rahul R" <rahul8590@gmail.com> wrote:
>>
>> I have a drpc server running in nimbus server.  After every 'X' minutes,
>> I see the following in the
>> drpc log
>>
>> ===
>> java.lang.OutOfMemoryError: Java heap space
>> at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57) ~[na:1.8.0_91]
>> at java.nio.ByteBuffer.allocate(ByteBuffer.java:335) ~[na:1.8.0_91]
>> at org.apache.thrift7.server.TNonblockingServer$FrameBuffer.
>> read(TNonblockingServer.java:491) ~[storm-core-0.9.6.jar:0.9.6]
>> at org.apache.thrift7.server.TNonblockingServer$SelectThread.
>> handleRead(TNonblockingServer.java:359) ~[storm-core-0.9.6.jar:0.9.6]
>> at org.apache.thrift7.server.TNonblockingServer$SelectThread.
>> select(TNonblockingServer.java:304) ~[storm-core-0.9.6.jar:0.9.6]
>> at org.apache.thrift7.server.TNonblockingServer$SelectThread.
>> run(TNonblockingServer.java:243) ~[storm-core-0.9.6.jar:0.9.6]
>> ====
>>
>>
>> There is enough memory available in the system .
>>
>> ====
>> storm@alerts:~/apache-storm-0.9.6/logs$ free -m
>>             total       used       free     shared    buffers     cached
>> Mem:          7985       2217       5767          0         81       1203
>> -/+ buffers/cache:        932       7052
>> Swap:            0          0          0
>> ====
>>
>>
>>
>> My DRPC Spout throws the following error
>>
>> =====
>> java.lang.RuntimeException: org.apache.thrift7.transport.TTransportException:
>> java.net.ConnectException: Connection refused at
>> backtype.storm.drpc.DRPCInvocationsClient.<init>(DRPCInvocationsClient.java:40)
>> at backtype.storm.drpc.DRPCSpout.open(DRPCSpout.java:87) at
>> backtype.storm.daemon.executor$fn*3371$fn*3386.invoke(executor.clj:529)
>> at backtype.storm.util$async_loop$fn__460.invoke(util.clj:461) at
>> clojure.lang.AFn.run(AFn.java:24) at java.lang.Thread.run(Thread.java:745)
>> Caused by: org.apache.thrift7.transport.TTransportException: java.net.ConnectException:
>> Connection refused at org.apache.thrift7.transport.TSocket.open(TSocket.java:183)
>> at org.apache.thrift7.transport.TFramedTransport.open(TFramedTransport.java:81)
>> at backtype.storm.drpc.DRPCInvocationsClient.connect(DRPCInvocationsClient.java:47)
>> at backtype.storm.drpc.DRPCInvocationsClient.<init>(DRPCInvocationsClient.java:38)
>> ... 5 more Caused by: java.net.ConnectException: Connection refused at
>> java.net.PlainSocketImpl.socketConnect(Native Method) at java.net
>> .AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at
>> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
>> at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
>> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net
>> .Socket.connect(Socket.java:589) at org.apache.thrift7.transport.TSocket.open(TSocket.java:178)
>> ... 8 more
>> =======
>>
>>
>> Restarting the DRPC Server solves the problem for the moment. Is there
>> any other better solution ?
>>
>>
>>
>> Thanks,
>> ./Rahul
>>
>>
>>
>>
>>
>
>

Mime
View raw message