storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tom Raney <tom.ra...@urbanairship.com>
Subject Re: Message processing from a kafka topic
Date Mon, 04 Jan 2016 21:13:24 GMT
Hi Birendra,

You didn't mention which version you are using, but in case it's current...

It looks like the zookeeper commit process submits the smallest pending
kafka offset per
https://github.com/apache/storm/blob/master/external/storm-kafka/src/jvm/storm/kafka/PartitionManager.java#L293.
So, in the event of a complete topology failure, you should have replay
beginning at the smallest unconfirmed offset for that broker/partition.

In the event the tuple is explicitly failed due to timeout, a re-delivery
attempt is made:
https://github.com/apache/storm/blob/master/external/storm-kafka/src/jvm/storm/kafka/PartitionManager.java#L179
.

Regards,

Tom

----- Original Message -----
From: Birendra Kumar Singh <singh.bk@gmail.com>
To: user@storm.apache.org
Cc:
Date: Sat, 2 Jan 2016 00:07:42 +0530
Subject: Message processing from a kafka topic
I have a storm setup that picks up messages from a kafka topic and
processes and persists them.
I want to understand how storm gurantees message processing in such a
scenario

Consider the below scenario:
I have configured multiple supervisors+workers for a storm cluster.
The KafkaSpout is reading message from the topic and then passes on this a
bolt. The bolt acks upon completion and the spout moves forward to the next
message.
I have 2 supervisors running - each of which are running 3 workers each.
>From what I understand - each of the worker on every supervisor is capable
to processing a message.
So, at any given time 6 messages are being processed parallely in storm
cluster.
what if the second message fails, either due to worker shutdown or due to
supervisor shutdown.
the zookeeper is already pointing to the 7 message for the consumer group.
In such a scenario, how will the second message get processed?

Thanks for reading
Birendra

Mime
View raw message