storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rahul R <rahul8...@gmail.com>
Subject Re: Re: [Storm v0.9.6] DRPC Server periodically crashes after sometime
Date Thu, 18 Aug 2016 05:19:22 GMT
I had cannot intentionally reproduce the problem. But it seems like,
doubling the memory given to drpc only stalls the OOM error for some more
time.  Which is the following.

----
java.lang.OutOfMemoryError: Java heap space
        at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
~[na:1.8.0_91]
        at java.nio.ByteBuffer.allocate(ByteBuffer.java:335) ~[na:1.8.0_91]
        at
org.apache.thrift7.server.TNonblockingServer$FrameBuffer.read(TNonblockingServer.java:491)
~[storm-core-0.9.6.jar:0.9.6]
        at
org.apache.thrift7.server.TNonblockingServer$SelectThread.handleRead(TNonblockingServer.java:359)
~[storm-core-0.9.6.jar:0.9.6]
        at
org.apache.thrift7.server.TNonblockingServer$SelectThread.select(TNonblockingServer.java:304)
~[storm-core-0.9.6.jar:0.9.6]
        at
org.apache.thrift7.server.TNonblockingServer$SelectThread.run(TNonblockingServer.java:243)
~[storm-core-0.9.6.jar:0.9.6]
2016-08-16T06:34:43.528+0000 o.a.t.s.TNonblockingServer [ERROR] run()
exiting due to uncaught error
------

I havent tinkered much with jvm OOM errors.  Any help/suggestions will be
much appreciated.

Thanks,
./Rahul



On Sat, Aug 13, 2016 at 6:13 AM, darion.yaphet <flykos@163.com> wrote:

> Could you reproduction this problem ?
>
> I guess you DRPC server message consume speed is slow than message product
> .
>
> jmap maybe could help you to pick out which object make OOM :)
>
> thanks
>
>
> 在 2016-08-11 14:17:46,"Rahul R" <rahul8590@gmail.com> 写道:
>
> Sure , thats one of the solutions. But I am not sure, whats causing DRPC
> to consume soo much of memory. Any suggestions of how to debug this ?
>
> Thanks,
> ./Rahul
>
> On Wed, Aug 10, 2016 at 6:24 PM, darion.yaphet <flykos@163.com> wrote:
>
>> Maybe you should check the JVM memory parameter -Xms and -Xmx to found
>> memory usage.
>>
>>
>>
>> At 2016-08-11 08:00:54, "Rahul R" <rahul8590@gmail.com> wrote:
>>
>> I have a drpc server running in nimbus server.  After every 'X' minutes,
>> I see the following in the
>> drpc log
>>
>> ===
>> java.lang.OutOfMemoryError: Java heap space
>> at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57) ~[na:1.8.0_91]
>> at java.nio.ByteBuffer.allocate(ByteBuffer.java:335) ~[na:1.8.0_91]
>> at org.apache.thrift7.server.TNonblockingServer$FrameBuffer.
>> read(TNonblockingServer.java:491) ~[storm-core-0.9.6.jar:0.9.6]
>> at org.apache.thrift7.server.TNonblockingServer$SelectThread.
>> handleRead(TNonblockingServer.java:359) ~[storm-core-0.9.6.jar:0.9.6]
>> at org.apache.thrift7.server.TNonblockingServer$SelectThread.
>> select(TNonblockingServer.java:304) ~[storm-core-0.9.6.jar:0.9.6]
>> at org.apache.thrift7.server.TNonblockingServer$SelectThread.
>> run(TNonblockingServer.java:243) ~[storm-core-0.9.6.jar:0.9.6]
>> ====
>>
>>
>> There is enough memory available in the system .
>>
>> ====
>> storm@alerts:~/apache-storm-0.9.6/logs$ free -m
>>             total       used       free     shared    buffers     cached
>> Mem:          7985       2217       5767          0         81       1203
>> -/+ buffers/cache:        932       7052
>> Swap:            0          0          0
>> ====
>>
>>
>>
>> My DRPC Spout throws the following error
>>
>> =====
>> java.lang.RuntimeException: org.apache.thrift7.transport.TTransportException:
>> java.net.ConnectException: Connection refused at
>> backtype.storm.drpc.DRPCInvocationsClient.<init>(DRPCInvocationsClient.java:40)
>> at backtype.storm.drpc.DRPCSpout.open(DRPCSpout.java:87) at
>> backtype.storm.daemon.executor$fn*3371$fn*3386.invoke(executor.clj:529)
>> at backtype.storm.util$async_loop$fn__460.invoke(util.clj:461) at
>> clojure.lang.AFn.run(AFn.java:24) at java.lang.Thread.run(Thread.java:745)
>> Caused by: org.apache.thrift7.transport.TTransportException: java.net.ConnectException:
>> Connection refused at org.apache.thrift7.transport.TSocket.open(TSocket.java:183)
>> at org.apache.thrift7.transport.TFramedTransport.open(TFramedTransport.java:81)
>> at backtype.storm.drpc.DRPCInvocationsClient.connect(DRPCInvocationsClient.java:47)
>> at backtype.storm.drpc.DRPCInvocationsClient.<init>(DRPCInvocationsClient.java:38)
>> ... 5 more Caused by: java.net.ConnectException: Connection refused at
>> java.net.PlainSocketImpl.socketConnect(Native Method) at java.net
>> .AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at
>> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
>> at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
>> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net
>> .Socket.connect(Socket.java:589) at org.apache.thrift7.transport.TSocket.open(TSocket.java:178)
>> ... 8 more
>> =======
>>
>>
>> Restarting the DRPC Server solves the problem for the moment. Is there
>> any other better solution ?
>>
>>
>>
>> Thanks,
>> ./Rahul
>>
>>
>>
>>
>>
>
>
>
>
>

Mime
View raw message