kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Abhinav Solan <abhinav.so...@gmail.com>
Subject Re: Kafka Consumer rebalancing frequently
Date Fri, 13 May 2016 14:20:47 GMT
Hi Sahitya,

Try reducing max.partition.fetch.bytes in your consumer.
Then also increase heartbeat.interval.ms, this might help in to delay the
consumer rebalance of your inbound process is taking more time than this

- Abhinav

On Fri, May 13, 2016 at 5:42 AM sahitya agrawal <sahitya2109@gmail.com>
wrote:

> Hi,
>
> I am using new Kafka consumer API ( 0.9.0.0) . I created 100 partitions  of
> a topic and started only one consumer to consume. Many of times , In
> consumer logs I see lot of rebalancing activity and no object is consumed
> due to that.
>
> Is this a known issue? Please let me know if some body can help with regard
> to this.
>
> My Consumer config:
>         props.put("zookeeper.session.timeout.ms", "10000");
>         props.put("rebalance.backoff.ms","10000");
>         props.put("zookeeper.sync.time.ms","200");
>         props.put("rebalance.max.retries","10");
>         props.put("enable.auto.commit", "false");
>         props.put("consumer.timeout.ms","20000");
>         props.put("auto.offset.reset", "smallest");
>
> Thanks,
> Sahitya
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message