storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From pradeep s <sreekumar.prad...@gmail.com>
Subject Storm kafka spout offset going back
Date Fri, 01 Sep 2017 05:04:12 GMT
Hi,
I am using Storm 1.1.0 ,storm kafka client version 1.1.1 and Kafka server
is 0.10.1.1.

Kakfa spout polling strategy used is UNCOMMITTED_EARLIEST.

Message flow is like below and its a normal topology

KafkaSpout --> AvroDeserializerBolt-->DataBaseInsertBolt.

If the message fails avro deserialization , i am moving the message to a
error queue and acknowledging from the avro bolt . This message is not
emitted to database bolt .

But its observed that after i restart topology , offset for the topic is
going back to old offset.

Will Kafka commit the offset, only if the message is acked from all bolts ?

Is the offset going back to previous value is beacuse of this ..

Thanks
Pradeep

Mime
View raw message