kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jay Kreps <jay.kr...@gmail.com>
Subject Re: Dynamically adding Kafka brokers
Date Mon, 03 Nov 2014 23:56:27 GMT
I agree it would be really nice to get KAFKA-1070 figured out.

FWIW, the reason for having a name or id other than ip was to make it
possible to move the identity to another physical server (e.g. scp the data
directory) and have it perform the same role on that new piece of hardware.
Systems that tie the data to a particular host tend to be sort of hated on
since you can't do anything simple/stupid to back them up or replace them.

-Jay

On Mon, Nov 3, 2014 at 2:23 PM, Gwen Shapira <gshapira@cloudera.com> wrote:

> +1
> Thats what we use to generate broker id in automatic deployments.
> This method makes troubleshooting easier (you know where each broker is
> running), and doesn't require keeping extra files around.
>
> On Mon, Nov 3, 2014 at 2:17 PM, Joe Stein <joe.stein@stealth.ly> wrote:
>
> > Most folks strip the IP and use that as the broker.id. KAFKA-1070 does
> not
> > yet accommodate for that very widely used method. I think it would be bad
> > if KAFKA-1070 only worked for new installations because that is how
> people
> > use Kafka today (per
> >
> >
> https://issues.apache.org/jira/browse/KAFKA-1070?focusedCommentId=14085808&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14085808
> > )
> >
> > On Mon, Nov 3, 2014 at 2:12 PM, Joel Koshy <jjkoshy.w@gmail.com> wrote:
> >
> > > KAFKA-1070 will help with this and is pending a review.
> > >
> > > On Mon, Nov 03, 2014 at 05:03:20PM -0500, Otis Gospodnetic wrote:
> > > > Hi,
> > > >
> > > > How do people handle situations, and specifically the broker.id
> > > property,
> > > > where the Kafka (broker) cluster is not fully defined right away?
> > > >
> > > > Here's the use case we have at Sematext:
> > > > * Our software ships as a VM
> > > > * All components run in this single VM, including 1 Kafka broker
> > > > * Of course, this is just for a nice OOTB experience, but to scale
> one
> > > > needs to have more instances of this VM, including more Kafka brokers
> > > > * *One can clone our VM and launch N instances of it, but because we
> > > have a
> > > > single Kafka broker config with a single broker.id <http://broker.id
> >
> > in
> > > > it, one can't just launch more of these VMs and expect to see more
> > Kafka
> > > > brokers join the cluster.  One would have to change the broker.id
> > > > <http://broker.id> on each new VM instance.*
> > > >
> > > > How do others handle this in a software that is packages and ships to
> > > user
> > > > and is not in your direct control to allow you to edit configs?
> > > >
> > > > Would it be best to have a script that connect to ZooKeeper to get
> the
> > > list
> > > > of all existing brokers and their IDs and then generate a new
> distinct
> > > ID +
> > > > config for the new Kafka broker?
> > > >
> > > > Or are there slicker ways to do this that people use?
> > > >
> > > > Thanks,
> > > > Otis
> > > > --
> > > > Monitoring * Alerting * Anomaly Detection * Centralized Log
> Management
> > > > Solr & Elasticsearch Support * http://sematext.com/
> > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message