kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stevo Slavić <ssla...@gmail.com>
Subject min ISR of 1 for consumer offsets topic and disabling unclean leader election - problem or not?
Date Sun, 10 Jan 2016 16:15:13 GMT
Hello Apache Kafka community,

Consumer offsets topic default configuration is somewhat different from all
other topics for a good reason, e.g. replication factor is 3 compared to 1
for other topics, and number of partitions is 50 compared to 1 for other
topic. Can somebody please explain why does consumer offsets topic default
to min.insync.replicas of 1, or in other words would it make sense to set
it to 2?

I guess that min.insync.replicas of 1 is preferred for higher availability
of consumer offsets topic partitions (whichever replica is alive allow it
to become leader regardless of how far behind it was) compared to higher
consistency across replicas achieved with min.insync.replicas of 2 or
higher.

Problem I see there, and please correct me if wrong, is that the
unclean.leader.election.enable seems to be configurable only on
cluster/broker level, one cannot override it on topic level (as documented
http://kafka.apache.org/documentation.html#topic-config ). On the other
hand one can override min.insync.replicas per topic and have it default to
1 on broker level.

So, if one disables unclean leader election, and has min.insync.replicas
set to 1 for consumer offsets topic and to 2 for some or all of the other
topics, one would actually make consumer offsets topic partitions less
available as loosing a lead would make partition unavailable until lead is
recovered, since no other replica if it wasn't part of ISR set cannot take
over, as by default with min.insync.replicas of 1 only a lead is guaranteed
to be part of ISR set.

Would it make sense to allow one to override unclean.leader.election.enable
on topic level, introduce separate
offsets.topic.unclean.leader.election.enable and have it default to true,
and introduce offsets.topic.min.insync.replicas defaulting to 1,
documenting rationale for all of these, suggesting one to increase
offsets.topic.min.insync.replicas if
offsets.topic.unclean.leader.election.enable is overridden to false?

Kind regards,
Stevo Slavic.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message