kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ivan mykhailov <ivan.mykhai...@strikead.com>
Subject Can current co-ordination protocol cause data loss or delays in consumption?
Date Thu, 10 Sep 2015 19:07:07 GMT
Hi,

we plan to use kafka as main message bus and we strongly rely on two
features:
- at-least-once delivery guarantee
- using groups for load balancing between several consumers

But there are some problems described in
https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Client+Re-Design
, like https://issues.apache.org/jira/browse/KAFKA-242

So questions are:
- can it lead to data loss?
-can it lead to delays in consumption? For example if rebalancing left some
partitions without assigned consumers for long time. Even if data is not
entirely lost, but just has not been consumed for about 10 minutes, it also
can break our processing.
- Due to roadmap
https://cwiki.apache.org/confluence/display/KAFKA/Future+release+plan new
client will be released in 0.8.3(Oct. 2015). Will it include new
co-ordination logic that fix this issue?

Kind Regards,

Ivan

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message