storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Birendra Kumar Singh <singh...@gmail.com>
Subject Message processing from a kafka topic
Date Fri, 01 Jan 2016 18:37:42 GMT
I have a storm setup that picks up messages from a kafka topic and
processes and persists them.
I want to understand how storm gurantees message processing in such a
scenario

Consider the below scenario:
I have configured multiple supervisors+workers for a storm cluster.
The KafkaSpout is reading message from the topic and then passes on this a
bolt. The bolt acks upon completion and the spout moves forward to the next
message.
I have 2 supervisors running - each of which are running 3 workers each.
>From what I understand - each of the worker on every supervisor is capable
to processing a message.
So, at any given time 6 messages are being processed parallely in storm
cluster.
what if the second message fails, either due to worker shutdown or due to
supervisor shutdown.
the zookeeper is already pointing to the 7 message for the consumer group.
In such a scenario, how will the second message get processed?

Thanks for reading
Birendra

Mime
View raw message