storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nitin Gupta <>
Subject RE: Topology gets stuck
Date Thu, 09 Jun 2016 14:18:20 GMT
Yes there are failing tuples.

The message timeout is set to 300 seconds using  the below configuration


From: Jungtaek Lim []
Sent: Thursday, June 09, 2016 7:41 PM
Subject: Re: Topology gets stuck


Could you check your configuration for message timeout seconds to see it's set to enough amount
of time, and also check there're failing tuples at that time?

Jungtaek Lim (HeartSaVioR)

2016년 6월 9일 (목) 오후 11:05, Nitin Gupta <<>>님이
Thanks Abhishek for the guidance.

As I mentioned the Bolt processes the same set of messages again and again and the consumer
offset doesn’t move ahead.
It processes around 100 messages , stops for few minutes (around 3-5 minutes)and then again
processes these messages. This cycle keeps repeating but the consumer offset in the zookeeper
is not updated.

I will check if it is possible to move to a new version of storm as the system is already
in production.

Thanks & Regards,
Nitin Gupta

From: Abhishek Agarwal [<>]
Sent: Thursday, June 09, 2016 7:26 PM

Subject: Re: Topology gets stuck

If spout is blocked only for a short duration, it may be due to slow bolt. If it remains blocked
forever, there is a possibility of deadlock. you can check out STORM-1027

On Thu, Jun 9, 2016 at 6:48 PM, Nitin Gupta <<>>
Hi Abhishek,

I am using version 0.9.4 .

From the log it seems the topology keeps processing a fix set of messages from the last offset
it got stuck . Looks it is equal to the max spout pending messages. It processes these messages
but doesn’t update the zookeeper with the processed offset. It then stop for a few minutes
and repeats the same process.

The rate at which the messages are being written is very high . So the lag is very high .
Not sure if the KafkaSpout stops processing after a specific number of messages pending to
be processed.

Thanks & Regards,
Nitin Gupta

From: Abhishek Agarwal [<>]
Sent: Thursday, June 09, 2016 6:40 PM
Subject: Re: Topology gets stuck

check the thread dump of your worker process after it gets stuck. which version do you use?

On Thu, Jun 9, 2016 at 6:36 PM, Nitin Gupta <<>>
Dear All,

I am using a Kafkaspout to process messages from Kafka. The bolt takes around 700 milliseconds
to process the message.
I observe the topology worked fine for a few hours but once the number of pending messages
increases it stops to update the consumer offset in the zookeeper.

I am facing this problem from quite few days. Any help will be highly appreciated.

Thanks & Regards,
Nitin Gupta

Abhishek Agarwal

Abhishek Agarwal

View raw message