storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nitin Gupta <>
Subject RE: Topology gets stuck
Date Thu, 09 Jun 2016 14:07:17 GMT
Thanks Abhishek for the guidance.

As I mentioned the Bolt processes the same set of messages again and again and the consumer
offset doesn’t move ahead.
It processes around 100 messages , stops for few minutes (around 3-5 minutes)and then again
processes these messages. This cycle keeps repeating but the consumer offset in the zookeeper
is not updated.

I will check if it is possible to move to a new version of storm as the system is already
in production.

Thanks & Regards,
Nitin Gupta

From: Abhishek Agarwal []
Sent: Thursday, June 09, 2016 7:26 PM
Subject: Re: Topology gets stuck

If spout is blocked only for a short duration, it may be due to slow bolt. If it remains blocked
forever, there is a possibility of deadlock. you can check out STORM-1027

On Thu, Jun 9, 2016 at 6:48 PM, Nitin Gupta <<>>
Hi Abhishek,

I am using version 0.9.4 .

From the log it seems the topology keeps processing a fix set of messages from the last offset
it got stuck . Looks it is equal to the max spout pending messages. It processes these messages
but doesn’t update the zookeeper with the processed offset. It then stop for a few minutes
and repeats the same process.

The rate at which the messages are being written is very high . So the lag is very high .
Not sure if the KafkaSpout stops processing after a specific number of messages pending to
be processed.

Thanks & Regards,
Nitin Gupta

From: Abhishek Agarwal [<>]
Sent: Thursday, June 09, 2016 6:40 PM
Subject: Re: Topology gets stuck

check the thread dump of your worker process after it gets stuck. which version do you use?

On Thu, Jun 9, 2016 at 6:36 PM, Nitin Gupta <<>>
Dear All,

I am using a Kafkaspout to process messages from Kafka. The bolt takes around 700 milliseconds
to process the message.
I observe the topology worked fine for a few hours but once the number of pending messages
increases it stops to update the consumer offset in the zookeeper.

I am facing this problem from quite few days. Any help will be highly appreciated.

Thanks & Regards,
Nitin Gupta

Abhishek Agarwal

Abhishek Agarwal

View raw message