kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joe Stein <joe.st...@stealth.ly>
Subject Re: Dynamically adding Kafka brokers
Date Mon, 03 Nov 2014 22:17:39 GMT
Most folks strip the IP and use that as the broker.id. KAFKA-1070 does not
yet accommodate for that very widely used method. I think it would be bad
if KAFKA-1070 only worked for new installations because that is how people
use Kafka today (per
https://issues.apache.org/jira/browse/KAFKA-1070?focusedCommentId=14085808&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14085808
)

On Mon, Nov 3, 2014 at 2:12 PM, Joel Koshy <jjkoshy.w@gmail.com> wrote:

> KAFKA-1070 will help with this and is pending a review.
>
> On Mon, Nov 03, 2014 at 05:03:20PM -0500, Otis Gospodnetic wrote:
> > Hi,
> >
> > How do people handle situations, and specifically the broker.id
> property,
> > where the Kafka (broker) cluster is not fully defined right away?
> >
> > Here's the use case we have at Sematext:
> > * Our software ships as a VM
> > * All components run in this single VM, including 1 Kafka broker
> > * Of course, this is just for a nice OOTB experience, but to scale one
> > needs to have more instances of this VM, including more Kafka brokers
> > * *One can clone our VM and launch N instances of it, but because we
> have a
> > single Kafka broker config with a single broker.id <http://broker.id> in
> > it, one can't just launch more of these VMs and expect to see more Kafka
> > brokers join the cluster.  One would have to change the broker.id
> > <http://broker.id> on each new VM instance.*
> >
> > How do others handle this in a software that is packages and ships to
> user
> > and is not in your direct control to allow you to edit configs?
> >
> > Would it be best to have a script that connect to ZooKeeper to get the
> list
> > of all existing brokers and their IDs and then generate a new distinct
> ID +
> > config for the new Kafka broker?
> >
> > Or are there slicker ways to do this that people use?
> >
> > Thanks,
> > Otis
> > --
> > Monitoring * Alerting * Anomaly Detection * Centralized Log Management
> > Solr & Elasticsearch Support * http://sematext.com/
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message