kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joseph Lawson <jlaw...@roomkey.com>
Subject Re: kafka cluster on aws
Date Wed, 14 Jan 2015 19:54:36 GMT
We have a separate daemon process that assigns EIPs to servers when they startup in an autoscaling
group based off of an autoscaling message.  So for a cluster of 3 we have 3 EIPs. Then we
inject the EIPs into startup script for Kafka which checks to see if it has one of the EIPs
and assigns itself the index of that IP so in the list:

1 is broker 0, 2 is broker 1 and 3 is broker 2.  All this is injected via cloudformation and
then we have a mod value so if we want to spin brokers in the same group we do mod 1,2 and
get brokers mod * 3 + index to determine which is in the group. (the EIPs are different as
it is a different cloudformation)

For redundancy make sure you run at least two that have full replicas of all other partitions.
 We run replication factor of 3 with three instances so if any goes down the other two bring
it back in sync once a fresh server spins in the autoscaling group.

From: Dillian Murphey <crackshotmule@gmail.com>
Sent: Wednesday, January 14, 2015 2:42 PM
To: users@kafka.apache.org
Subject: kafka cluster on aws

I can't seem to find much information to help me (being green to kafka) on
setting up a cluster on aws. Does anyone have any sources?

The question I have off the bat is, what methods have already been explored
to generate a unique broker id? If I spin up a new server, do I just need
to maintain my own broker-id list somewhere so I don't re-use an already
allocated broker id?

Also, I read an article about a broker going down and requiring a new
broker be spun up with the same id. Is this also something I need to

I want to setup a kafka auto-scaling group on AWS, so I can add brokers at
well or based on load. It doesn't seem too complicated, or maybe I'm too
green to see it, but I don't want to re-invent everything myself.

I know Loggly uses AWS/Kafka, so I'm hunting for more details on that too.

Thanks for any help
View raw message