kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aditya Auradkar <aaurad...@linkedin.com.INVALID>
Subject RE: batching causes replica out of sync
Date Fri, 06 Mar 2015 02:46:20 GMT

Just FYI - Here's a discussion on this issue if you are interested.

From: Mayuresh Gharat [gharatmayuresh15@gmail.com]
Sent: Thursday, March 05, 2015 4:41 PM
To: users@kafka.apache.org
Subject: Re: batching causes replica out of sync

Whats the size of the batch that you are sending?
If the messages produced are more that 4000 at any point of time there is a
chance of the replica falling behind more than 4000 messages and being
kicked out of the ISR. This happens because the thread that checks for in
sync replicas is asynchronous. The replica gets added back once it catches
up. We at Linkedin have seen this in our test clusters.

If this happens and you loose the leader broker at this point you will have
unclean leader election. If this is the case you should increase the
replica.max.lag.message parameter value.



On Thu, Mar 5, 2015 at 2:50 PM, xiaoyu wang <xiaoyu.wang@gmail.com> wrote:

> Hi all,
> We previously have replica.max.lag.message set to 4000 and use sync
> producer to send data to kafka, one message at a time. With this, we don't
> see many unclean leader election.
> Recently, we switched to use sync producer and batch messages. After that,
> we see unclean leader election more often. Did you observe the same?
> Do you have recommendations about how broker configuration should be
> adjusted based on the batch size?
> Thanks,

Mayuresh R. Gharat
(862) 250-7125

View raw message