kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Florian Dambrine <flor...@gumgum.com>
Subject Re: Dynamically adding Kafka brokers
Date Mon, 03 Nov 2014 22:09:43 GMT
I am automating Kafka clusters in the cloud and I am using Ips (stripped to
keep only numbers) to define the broker ids.

You might have to add a shell script or whatever fits to set the broker id.

I hope this might help a bit.

Regards,
Le 3 nov. 2014 23:04, "Otis Gospodnetic" <otis.gospodnetic@gmail.com> a
écrit :

> Hi,
>
> How do people handle situations, and specifically the broker.id property,
> where the Kafka (broker) cluster is not fully defined right away?
>
> Here's the use case we have at Sematext:
> * Our software ships as a VM
> * All components run in this single VM, including 1 Kafka broker
> * Of course, this is just for a nice OOTB experience, but to scale one
> needs to have more instances of this VM, including more Kafka brokers
> * *One can clone our VM and launch N instances of it, but because we have a
> single Kafka broker config with a single broker.id <http://broker.id> in
> it, one can't just launch more of these VMs and expect to see more Kafka
> brokers join the cluster.  One would have to change the broker.id
> <http://broker.id> on each new VM instance.*
>
> How do others handle this in a software that is packages and ships to user
> and is not in your direct control to allow you to edit configs?
>
> Would it be best to have a script that connect to ZooKeeper to get the list
> of all existing brokers and their IDs and then generate a new distinct ID +
> config for the new Kafka broker?
>
> Or are there slicker ways to do this that people use?
>
> Thanks,
> Otis
> --
> Monitoring * Alerting * Anomaly Detection * Centralized Log Management
> Solr & Elasticsearch Support * http://sematext.com/
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message