kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From François Méthot <fmetho...@gmail.com>
Subject KafkaSpout forceFromStart Issue
Date Mon, 23 Mar 2015 18:16:31 GMT
Hi,

  We have a storm topology that uses Kafka to read a topic with 6
partitions. (  Kafka 0.8.2, Storm 0.9.3 )

Recently, we had to set the KafkaSpout to read from the beginning, so we
temporary configured our KafkaConfig this way:

kafkaConfig.forceFromStart=true
kafkaConfig.startOffsetTime = OffsetRequest.EarliestTime()

It worked well, but afterward, setting those parameters back to false and
to LatestTime respectively had no effect. In fact the topology won't read
from our topic anymore.

When the topology starts, The spout successully logs the offset and
consumer group's cursor position for each partition in the worker log. But
nothing is read.

The only way we can read back from our Topic is to give our SpoutConfig a
new Kafka ConsumerGroup Id.

So it looks like, if we don't want to modify any KafkaSpout/Kafka code, the
only way to read from the beginning would be to write the position we want
to read from in Zookeeper for our Consumer Group where offset are stored
and to restart our topology.
Would anyone know if this is a bug in the KafkaSpout or an issue inherited
from bug in Kafka?
Thanks
Francois

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message