kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Seshadri, Balaji" <Balaji.Sesha...@dish.com>
Subject commitOffsets by partition 0.8-beta
Date Fri, 18 Apr 2014 17:02:22 GMT

We have use case in DISH where we need to stop the consumer when we have issues in proceeding
further to database or another back end.

We update offset manually for each consumed message. There are 4 threads(e.g) consuming from
same connector and when one thread commits the offset there is chance that data for all other
threads also get committed.

We don't want to go with this to prod as we are going to take first step of replacing traditional
broker with Kafka for business critical process, is it ok if we add commit Offset(Topic,partition)
method that commits only the consumed data for that particular thread.

At this point we don't want to change our framework to use Simple Consumer as it is lots of
work for us.

Please let us know the effect of committing the offset per partition being consumed by the
thread. We have around 131 partitions per topic and around  20 topics.



  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message