kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mayuresh Gharat <gharatmayures...@gmail.com>
Subject Re: Optimising Time For Leader Election
Date Thu, 06 Dec 2018 23:26:19 GMT
Hi Mark,

The leader election of a new topic partition happens once the controller
detects that the Leader has crashed.
This happens asynchronously via a zookeeper listener. Once a zookeeper
listener is fired, the corresponding object indicating the event happened
is put in to a controller queue.
The controller has a single thread that pulls data out of this queue and
handles each event one after another.
I can't remember of a config to tune this, on top of my head.
How much delay are you seeing in leadership change? Are there any
controller socket timeouts in the log?
Also might want to take a look at KIP-291 (KAFKA-4453), which is meant for
shortening this time period for handling controller events.



On Thu, Dec 6, 2018 at 9:50 AM Harper Henn <harper.henn@datto.com> wrote:

> Hi Mark,
> If a broker fails and you want to elect a new leader as quickly as
> possible, you could tweak zookeeper.session.timeout.ms in the kafka broker
> configuration. According to the documentation: "If the consumer fails to
> heartbeat to ZooKeeper for this period of time it is considered dead and a
> rebalance will occur."
> https://kafka.apache.org/0101/documentation.html
> I think making zookeeper.session.timeout.ms smaller will result in faster
> detection of a dead node, but the downside is that a leader election might
> get triggered by network blips or other cases where your broker is not
> actually dead.
> Harper
> On Thu, Dec 6, 2018 at 9:11 AM Mark Anderson <manderson23@gmail.com>
> wrote:
> > Hi,
> >
> > I'm currently testing how Kafka reacts in cases of broker failure due to
> > process failure or network timeout.
> >
> > I'd like to have the election of a new leader for a topic partition
> happen
> > as quickly as possible but it is unclear from the documentation or broker
> > configuration what the key parameters are to tune to make this possible.
> >
> > Does anyone have any pointers? Or are there any guides online?
> >
> > Thanks,
> > Mark
> >

Mayuresh R. Gharat
(862) 250-7125

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message