Hi Abhishek,

 

I am using version 0.9.4 . 

 

From the log it seems the topology keeps processing a fix set of messages from the last offset it got stuck . Looks it is equal to the max spout pending messages. It processes these messages but doesn’t update the zookeeper with the processed offset. It then stop for a few minutes and repeats the same process.

 

The rate at which the messages are being written is very high . So the lag is very high . Not sure if the KafkaSpout stops processing after a specific number of messages pending to be processed.

 

Thanks & Regards,

Nitin Gupta

 

From: Abhishek Agarwal [mailto:abhishcool@gmail.com]
Sent: Thursday, June 09, 2016 6:40 PM
To: user@storm.apache.org
Subject: Re: Topology gets stuck

 

check the thread dump of your worker process after it gets stuck. which version do you use?

 

On Thu, Jun 9, 2016 at 6:36 PM, Nitin Gupta <Nitin.Gupta@e-zest.in> wrote:

Dear All,

 

I am using a Kafkaspout to process messages from Kafka. The bolt takes around 700 milliseconds to process the message.

I observe the topology worked fine for a few hours but once the number of pending messages increases it stops to update the consumer offset in the zookeeper.

 

I am facing this problem from quite few days. Any help will be highly appreciated.

 

Thanks & Regards,

Nitin Gupta

 



 

--

Regards,
Abhishek Agarwal